Method and device for automatically identifying risk content
Through the construction of multimodal content understanding and risk content recognition tasks, the risk content is automatically identified, and the problems of low recall rate and low manual analysis efficiency in the existing technology are solved, and rapid and accurate risk content recognition and automated processing are achieved.
Patent Information
- Application Number
- CN202411659667.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-19
- Publication Date
- 2025-03-28
AI Technical Summary
The prior art has poor recall rates when identifying risky content and requires manual analysis, resulting in inefficiency and waste of labor.
By obtaining risk content description information, multi-modal content understanding, generating target risk text or pictures, and building risk content recognition tasks, querying content search systems to obtain recall content, and determining the discrimination result of recall content through the discrimination unit.
It realizes rapid and accurate automatic identification of risk content, reduces manual intervention, improves efficiency and reduces labor costs.
Smart Images
Figure CN119719457A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and more specifically, to a method and device for automatically identifying risky content. Background Art
[0002] With the development of technology, various shopping platforms, information dissemination platforms, etc. are becoming more and more integrated into users' lives and work. Various information will also be transmitted to users through the above-mentioned shopping platforms or information dissemination platforms, including some negative information, which can also be called risky content. Therefore, each platform needs to identify risky content and reduce users' exposure to negative information.
[0003] In the prior art, a pre-trained content risk identification model is used to search for risky content on various platforms. However, since risky content is changing, the sample size of each risky content is relatively small, and a content risk identification model cannot be trained separately for each risky content. The recall rate of a general content risk identification model is poor. In cases where a quick response is required, manual analysis of the risky content is required, and then the risky content is searched on various platforms, and manual judgment is required as to whether the recalled content is correct.
[0004] To sum up, how to automatically identify risky content quickly, accurately and without wasting manpower is a problem that needs to be solved at present. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a method and device for automatically identifying risky content, which can automatically identify risky content on the basis of being fast, accurate and without wasting manpower.
[0006] In a first aspect, an embodiment of the present invention provides a method for automatically identifying risky content, the method comprising:
[0007] Obtain risk content description information;
[0008] Performing multimodal content understanding on the risk content description information to generate at least one target risk text or target risk picture;
[0009] Perform risk understanding on the at least one target risk text, the target risk picture, or the risk content description information, and construct at least one risk content identification task;
[0010] querying a content search system according to the at least one risk content identification task to obtain N recalled contents, wherein the recalled contents are contents related to the risk content description information, the target risk text or the target risk picture, and N is a positive integer greater than or equal to 1;
[0011] Determine the judgment result of each of the recalled contents.
[0012] Optionally, the method further includes:
[0013] The determination result is output as at least one recall content that has a risk.
[0014] Optionally, the method further includes:
[0015] In response to the judgment result that the N recalled contents have risks, the recalled contents with the judgment result that the risks exist are added to the risk content description information, and the multimodal content understanding, the risk understanding and subsequent processing processes are continued.
[0016] Optionally, the method further includes:
[0017] In response to the judgment result that there is no risk in the N recalled contents, the recalled contents with the judgment result that there is no risk are added to the risk content description information, and the multimodal content understanding, risk understanding and subsequent processing processes are continued, wherein the recalled contents that do not have risk are negative feedback.
[0018] Optionally, the method further includes:
[0019] In response to the number of times the recalled content is obtained being less than or equal to a set threshold, the next query of the content search system is stopped.
[0020] Optionally, the method further includes:
[0021] In response to the risk content description information being audio information or video information, preprocessing the audio information or video information to generate text information or picture information;
[0022] The multimodal content understanding of the risk content description information specifically includes:
[0023] The text information or picture information is subjected to multimodal content understanding.
[0024] Optionally, the performing multimodal content understanding on the text information or the picture information specifically includes:
[0025] Using a first text model to understand the text content of the text information, wherein the first text model is a large language model fine-tuned using historical risk data, and is used to extract target risk text from the text information;
[0026] A first image model is used to understand the image content of the image information, wherein the first image model is a multimodal model fine-tuned using historical risk data and is used to extract target risk images from the image information.
[0027] Optionally, performing risk understanding on the at least one target risk text or the target risk picture to construct at least one risk content identification task specifically includes:
[0028] Using a second text model to understand the target risk text, and construct at least one risk content identification task, wherein the second text model is a large language model that uses historical risk data for supervised fine-tuning and preference alignment, and is used to search for a target risk text suitable for constructing a risk content identification task among multiple target risk texts;
[0029] A second image model is used to understand the risk of the target risk image and construct at least one risk content identification task, wherein the second image model is a multimodal model that uses historical risk data for supervised fine-tuning and preference alignment, and is used to search for a target risk image suitable for constructing a risk content identification task among multiple target risk images.
[0030] Optionally, querying a content search system according to the at least one risk content identification task to obtain N recalled contents specifically includes:
[0031] Build the task search process based on the LangGraph framework;
[0032] The at least one risk content identification task queries the content search system according to the task search process to obtain the N recalled contents.
[0033] In a second aspect, an embodiment of the present invention provides a device for automatically identifying risky content, the device comprising:
[0034] An acquisition unit, used to acquire risk content description information;
[0035] A generating unit, configured to perform multimodal content understanding on the risk content description information to generate at least one target risk text or target risk picture;
[0036] A construction unit, configured to perform risk understanding on the at least one target risk text, or the target risk picture, or the risk content description information, and construct at least one risk content identification task;
[0037] The acquisition unit is further used to query the content search system according to the at least one risk content identification task to acquire N recalled contents, wherein the recalled contents are contents related to the target risk text or target risk picture, and N is a positive integer greater than or equal to 1;
[0038] A determination unit is used to determine a determination result of each of the recalled contents.
[0039] Optionally, the device further comprises:
[0040] The output unit is used to output at least one recall content that is judged to have a risk.
[0041] Optionally, the acquisition unit is further used for:
[0042] In response to the judgment result that the N recalled contents have risks, the recalled contents with the judgment result that the risks exist are added to the risk content description information, and the multimodal content understanding, the risk understanding and subsequent processing processes are continued.
[0043] Optionally, the acquisition unit is further used for:
[0044] In response to the judgment result that there is no risk in the N recalled contents, the recalled contents with the judgment result that there is no risk are added to the risk content description information, and the multimodal content understanding, risk understanding and subsequent processing processes are continued, wherein the recalled contents that do not have risk are negative feedback.
[0045] Optionally, the acquisition unit is further used for:
[0046] In response to the number of times the recalled content is obtained being less than or equal to a set threshold, the next query of the content search system is stopped.
[0047] Optionally, the generating unit is further used for:
[0048] In response to the risk content description information being audio information or video information, preprocessing the audio information or video information to generate text information or picture information;
[0049] The generating unit is specifically used for:
[0050] The text information or picture information is subjected to multimodal content understanding.
[0051] Optionally, the generating unit is specifically used for:
[0052] Using a first text model to understand the text content of the text information, wherein the first text model is a large language model fine-tuned using historical risk data, and is used to extract target risk text from the text information;
[0053] A first image model is used to understand the image content of the image information, wherein the first image model is a multimodal model fine-tuned using historical risk data and is used to extract target risk images from the image information.
[0054] Optionally, the construction unit is specifically used for:
[0055] Using a second text model to understand the target risk text, and construct at least one risk content identification task, wherein the second text model is a large language model that uses historical risk data for supervised fine-tuning and preference alignment, and is used to search for a target risk text suitable for constructing a risk content identification task among multiple target risk texts;
[0056] A second image model is used to understand the risk of the target risk image and construct at least one risk content identification task, wherein the second image model is a multimodal model that uses historical risk data for supervised fine-tuning and preference alignment, and is used to search for a target risk image suitable for constructing a risk content identification task among multiple target risk images.
[0057] Optionally, the acquiring unit is specifically used for:
[0058] Build the task search process based on the LangGraph framework;
[0059] The at least one risk content identification task queries the content search system according to the task search process to obtain the N recalled contents.
[0060] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in the first aspect or any possible embodiment of the first aspect.
[0061] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a method as described in the first aspect or any possible embodiment of the first aspect.
[0062] In an embodiment of the present invention, risk content description information is obtained; multimodal content understanding is performed on the risk content description information to generate at least one target risk text or target risk picture; risk understanding is performed on the at least one target risk text or target risk picture to construct at least one risk content identification task; a content search system is queried according to the at least one risk content identification task to obtain N recalled contents, wherein the recalled contents are contents related to the target risk text or target risk picture, and N is a positive integer greater than or equal to 1; and the discrimination result of each recalled content is determined. Through the above method, the recalled contents can be automatically identified quickly and accurately without wasting manpower, and the risky contents in the recalled contents can be determined. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
[0064] Figure 1 is a schematic diagram of the structure of an intelligent agent in an embodiment of the present invention;
[0065] Figure 2 is a flow chart of a method for automatically identifying risky content in an embodiment of the present invention;
[0066] Figure 3 is a flow chart of a method for automatically identifying risky content in an embodiment of the present invention;
[0067] Figure 4 is a schematic diagram of a device for automatically identifying risky content in an embodiment of the present invention;
[0068] Figure 5 is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION
[0069] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.
[0070] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.
[0071] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".
[0072] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.
[0073] In the prior art, risky content can also be called illegal content, which is usually presented by user reports, regulatory instructions or external public opinion. Each platform needs to set up a corresponding risk content prevention and control system to identify risky content. Specifically, a pre-trained content risk identification model is used to search for risky content in each platform. However, since risky content changes with the changes in the above-mentioned user reports, regulatory instructions or external public opinion, the sample size of each risky content is relatively small, which is not enough to train a content risk identification model for each risky content separately, and the recall rate of the general content risk identification model is poor; in the case of rapid response, it is also necessary to manually analyze the risky content, for example, a large amount of manpower is used to analyze the risky content one by one, extract the query words or query images that need to be checked, and then search in each platform according to the above query words or query images to obtain the recalled content, and then manually judge whether the recalled content is correct, that is, judge whether the recalled content has risks or violations. Therefore, how to automatically identify risky content quickly and accurately without wasting manpower is a problem that needs to be solved at present.
[0074] In the embodiment of the present invention, in order to solve the above problems, an intelligent agent is designed, and the structural diagram of the intelligent agent is as follows: Figure 1 As shown, it includes a content understanding module 101, a task planning module 102 and an evaluation module 103, wherein after the intelligent agent receives risk content such as user reports, supervisory instructions or external public opinion, the task planning module 102 calls the content understanding module 101 to process the risk content, extracts the information to be queried, and constructs a risk content identification task according to the information to be queried. After receiving the risk content identification task, the task planning module 102 calls the interface tool corresponding to the content search system, and indexes the content through the risk content identification task, retrieves multiple recalled contents in the content search system, and then the task planning module 102 calls the evaluation module 103 to evaluate the multiple recalled contents, and after determining the risky content, sends it to the user through the intelligent agent.
[0075] In a possible implementation, based on the intelligent agent, a method for automatically identifying risky content is proposed, specifically: Figure 2 As shown, the method includes:
[0076] Step S201, obtaining risk content description information.
[0077] Specifically, the risk content description information is user reports, supervisory instructions or external public opinion, etc. The risk content description information includes text, pictures, audio, video and various types of attachment content. The attachment content may also include text, pictures, audio or video.
[0078] For example, suppose the supervisory instruction is to supervise a certain word that makes the user feel negative emotions, and a specific example is given in the attachment of the supervisory instruction.
[0079] In a possible implementation, the task planning module obtains risk content description information input from the outside.
[0080] Step S202: Perform multimodal content understanding on the risk content description information to generate at least one target risk text or target risk picture.
[0081] In a possible implementation, the risk content description information includes text information or picture information, and the task planning module calls the content understanding module, and uses the text understanding module in the content understanding module to perform multimodal content understanding on the risk content description information, specifically including:
[0082] A first text model is used to understand the text content of the text information, wherein the first text model is a large language model fine-tuned using historical risk data, and is used to extract target risk text from the text information; or a first picture model is used to understand the image content of the picture information, wherein the first picture model is a multimodal model fine-tuned using historical risk data, and is used to extract target risk images from the picture information.
[0083] In the embodiment of the present invention, the target risk text is a specific word or code (Identity Document, ID), and the ID includes a user ID, a product ID, a store ID, etc.
[0084] In one possible implementation, the multimodal content understanding of the text information can also be called text content understanding. Since the source of risk content description information is relatively wide, the risk content description information contains many complex documents, such as instruction background, instruction requirements, instruction legality, intelligence sources, whistleblower dissatisfaction expression, etc., various text contents related to or unrelated to the subject of the risk content description information need to be extracted through the first text model to extract the subject and subject-related text content, and integrate the content of multiple files while ensuring that core information is not lost, wherein the subject is the key subject of the risk content, that is, the target risk text generated after text content understanding.
[0085] In a possible implementation, the text content understanding also includes merging repeated content. In the attachments of various data sources, the example sample descriptions in the same attachment are often repeated, but the content ID in each sample description is different. Therefore, the repeated content needs to be merged to reduce the difficulty of subsequent risk understanding; and in various source data, there are often special texts such as Uniform Resource Locator (URL), image URL, content ID, user ID, user nickname (nick) corresponding to the content, so special text content extraction is required. First, the actual content ID or the content itself is obtained through the URL, and then various ID, nick and other information are extracted from the overall text to facilitate subsequent processing; because the subject may be a concept, event or organization and other information, but its related content may not be in the training data of the first text model, so the first text model may not understand the actual meaning. By using an external search engine to expand the content, the first text model can better generate the target risk text to facilitate subsequent processing.
[0086] In one possible implementation, the multimodal content understanding of the image information may also be referred to as image content understanding. Since the source of risk content description information is relatively wide, the text paragraphs in the risk content description information or in attachments may contain one or more image contents, some of which are related to the subject, and some of which are only used as additional explanations or to ensure the legality of the program. The subject and images related to the subject need to be extracted through the first image model, wherein the subject is the key subject of the risk content, that is, the target risk image generated after image content understanding.
[0087] In one possible implementation, since the image content to be recalled does not provide an image example but only a text description, it is necessary to use an external search engine to obtain the specific image content. The above method can effectively improve the recall rate, that is, the proportion of valid content in the recalled content.
[0088] In one possible implementation, since the risk content description information also includes audio or video, the attachments of various data sources may contain one or more audio and / or video content, some of which are related to the subject, and some of which are only used for additional explanations, to ensure the legality of the program, etc. It is necessary to extract the audio and / or video content related to the subject through a multimodal model, generate text information or image information based on the audio or video, and further perform multimodal content understanding through a first text model or a first image model to ensure that other irrelevant content does not enter the subsequent processing flow.
[0089] In one possible implementation, data from various sources may be accompanied by one or more attachments for supplementing some additional information. The file types of attachments vary. For text, picture, audio, and video files, multimodal content understanding can be performed according to the above method; for attachments in other formats, such as zip, doc, xlsx and other formats, the content in the attachment needs to be converted into a lightweight markup language (markdown) format, and then the content can be understood according to the above method.
[0090] In a possible implementation, the multimodal content understanding module returns data to the task planning module. For example, the returned content is as follows:
[0091] "taskType":["recognition"]
[0092] "text":["XXXXXXX","YYYYYYY"...],
[0093] "picture":["pic_path1","pic_path2",...],
[0094] "audio":["audio_path1","audio_path2",...],
[0095] "video":["video_path1","video_path2",...],
[0096] "contentID":["id1","id2",...],
[0097] "userID":["userid1","userid2"...]
[0098] }
[0099] Among them, the "taskType" is the task type. If the task type is "recognition", that is, recognition, which can also be called risk recognition, the task planning module plans the next risk understanding task and query task based on the content returned by the multimodal content understanding module; the returned content may include specific vocabulary, picture path, audio path, video path, specific ID, etc. generated by multimodal content understanding for text, images, audio, video, content ID, user ID, etc.
[0100] Step S203: perform risk understanding on the at least one target risk text, the target risk picture, or the risk content description information, and construct at least one risk content identification task.
[0101] Specifically, a second text model is used to understand the risk of the target risk text and construct at least one risk content identification task, wherein the second text model is a large language model that uses historical risk data for supervised fine-tuning (instruction Supervised Fine-Tuning, instruction SFT) and preference alignment (preference Alignmen), and is used to search for target risk texts suitable for constructing risk content identification tasks among multiple target risk texts; or, a second picture model is used to understand the risk of the target risk text and construct at least one risk content identification task, wherein the second picture model is a multimodal model that uses historical risk data for supervised fine-tuning and preference alignment, and is used to search for target risk pictures suitable for constructing risk content identification tasks among multiple target risk pictures.
[0102] In a possible implementation, the intelligent agent may directly understand the risk content description information.
[0103] In one possible implementation, for audio, Automatic Speech Recognition (ASR) is first used to convert the audio into text, and the text content and risk understanding are performed on the text to construct a risk content identification task; for video, a key frame extraction model is first used to convert a video into a picture list, and then the picture content and risk understanding are performed on each picture in the picture list to construct a risk content identification task.
[0104] In a possible implementation, the data output by the risk understanding module for risk understanding is sent to the task planning module for subsequent planning. The specific content sent is as follows:
[0105] {
[0106] "text_query":["XXXXXXX","YYYYYYY"...],
[0107] "pic_query":["pic_path1","pic_path2",...],
[0108] "audio_query":["aaaaaa","bbbbbb",...],
[0109] "video_query":["video_pic_path1","video_pic_path2",...],
[0110] "contentID":["id1","id2",...],
[0111] "userID":["userid1","userid2"...]
[0112] }
[0113] The returned content may include text, images, audio, video, content ID, user ID, etc.
[0114] In a possible implementation, the risk understanding module may not be able to understand the risk content description information when understanding it. After obtaining new risk content through the task planning module, multimodal content understanding and risk understanding can be performed again.
[0115] Step S204: query the content search system according to the at least one risk content identification task to obtain N recalled contents.
[0116] Among them, the recall content is the risk content description information, the target risk text or the content related to the target risk picture, and N is a positive integer greater than or equal to 1.
[0117] Specifically, the task planning module calls the interface tool of the content search system according to the risk content identification task, and the tool interface types include text query, image query, image expansion source, ID query, etc.; the task planning module is specifically a large language model that is fine-tuned by historical risk data. The historical risk data is processed from the operation log data of manual risk investigation. The operation log data contains various types of search actions, search display content, whether the displayed content is illegal, and the next query action, etc., which are determined according to the actual situation.
[0118] In a possible implementation, the task planning module constructs a task search process according to the LangGraph framework in task planning, and then the at least one risk content identification task queries the content search system according to the task search process to obtain the N recalled contents.
[0119] Specifically, the task search process constructed according to the LangGraph framework includes: generating nodes, generating edges and binding states, wherein each node represents a specific action or decision point. When creating a node, two key information needs to be configured, namely, the node name and the node execution method. The node name concisely and clearly summarizes the action represented by the node to facilitate the understanding of the overall process; the node execution method specifically indicates the operation content that the large language model LLM needs to perform, for example, calling an interface. Assume that in the task of checking the goods related to "water cup", a node named "text check_water cup" can be created. The execution method is to call the text check API, pass the parameter "water cup", check the text information of each dimension of the goods, and return the return result to the subsequent node. In the execution graph corresponding to the task search process, the edge is used to connect two adjacent nodes, indicating the process from one node to another; the edge is directed, that is, from the upstream node to the downstream node; in certain cases, it may be necessary to set conditions between the upstream and downstream nodes of the edge. Only when these conditions are met, the task search process will be executed from the upstream node to the downstream node. During the task planning process, the execution method of each node needs to receive a state parameter and write the returned result into the state variable so that the subsequent executed nodes can understand the current context and make correct decisions.
[0120] Step S205: Determine the identification result of each of the recalled contents.
[0121] Specifically, each of the N recalled contents is judged by the evaluation module to determine whether it has risks.
[0122] In a possible implementation, based on the identification result of the recalled content, the next query task is planned until the risk content related to the initial risk content description information cannot be identified after iteration, or after iterating a set number of times, the query can be stopped.
[0123] In one possible implementation, in response to a judgment result that a set proportion or a set number of the N recalled contents is that there is a risk, the N recalled contents are added to the risk content description information, and the multimodal content understanding and subsequent processing process is continued; or, in response to a judgment result that a set proportion or a set number of the N recalled contents is that there is no risk, the N recalled contents are added to the risk content description information as negative feedback, and the multimodal content understanding and subsequent processing process is continued; or, in response to the number of times the recalled contents are obtained being less than a set threshold, it is determined whether to perform the next retrieval according to the proportion or number of risky contents in the recalled contents, until the next query of the content search system is stopped in response to the number of times the recalled contents are obtained being equal to the set threshold; or, in response to the fact that the content search system has been fully searched after obtaining the recalled contents, the next query of the content search system is stopped.
[0124] In a possible implementation, the evaluation module is a large language model aligned with historical risk data.
[0125] Through the above method, multiple rounds of queries are performed on the content search system to achieve automatic identification and query of risky content.
[0126] In a possible implementation, after step S205, other steps are also included, specifically: Figure 3 As shown, including the following:
[0127] Step S206: outputting at least one recall content that is judged to have a risk.
[0128] Specifically, after the task planning module determines that the task is completed, the display interface of the agent displays the at least one recall content that is judged to have a risk, and the output result is as follows:
[0129]
[0130]
[0131] Among them, the "text query content" and "multimodal query content" belong to the risk understanding, and the specific return content is determined according to the actual situation. This is only an exemplary description.
[0132] In an embodiment of the present invention, a specific execution process of the task planning module is described in detail through a specific flow chart, specifically including: the task planning module first calls the content understanding module to perform multimodal content understanding and risk understanding, and then calls the interface tool to perform image query, ID query and text query, and summarizes the queried recall content, wherein the called interface tool is variable and is determined according to the risk content identification task constructed according to the multimodal content understanding and risk understanding.
[0133] In the embodiment of the present invention, the above Figure 4 The schematic diagram for single-round task planning is for illustrative purposes only.
[0134] Through the above embodiments, the task planning module and the evaluation module in the intelligent agent can change the query strategy at an appropriate time and determine when to terminate the query task, thereby realizing multiple rounds of queries without human intervention, thereby realizing automatic query and identification of risk content similar to the input risk case, improving the response speed to the risk content, and reducing labor costs.
[0135] In an embodiment of the present invention, a device for automatically identifying risky content is provided, such as Figure 4 As shown, it specifically includes: an acquisition unit 401, a generation unit 402, a construction unit 403 and a discrimination unit 404; wherein, the acquisition unit 401 is used to acquire risk content description information; the generation unit 402 is used to perform multimodal content understanding on the risk content description information, and generate at least one target risk text or target risk picture; the construction unit 403 is used to perform risk understanding on the at least one target risk text, or the target risk picture, or the risk content description information, and construct at least one risk content identification task; the acquisition unit 401 is also used to query the content search system according to the at least one risk content identification task, and obtain N recalled contents, wherein the recalled contents are contents related to the target risk text or target risk picture, and N is a positive integer greater than or equal to 1; the discrimination unit 404 is used to determine the discrimination result of each of the recalled contents.
[0136] Furthermore, the device also includes:
[0137] The output unit is used to output at least one recall content that is judged to have a risk.
[0138] Furthermore, the acquisition unit is also used for:
[0139] In response to the judgment result that the N recalled contents have risks, the recalled contents with the judgment result that the risks exist are added to the risk content description information, and the multimodal content understanding, the risk understanding and subsequent processing processes are continued.
[0140] Furthermore, the acquisition unit is also used for:
[0141] In response to the judgment result that there is no risk in the N recalled contents, the recalled contents with the judgment result that there is no risk are added to the risk content description information, and the multimodal content understanding, risk understanding and subsequent processing processes are continued, wherein the recalled contents that do not have risk are negative feedback.
[0142] Furthermore, the acquisition unit is also used for:
[0143] In response to the number of times the recalled content is obtained being less than or equal to a set threshold, the next query of the content search system is stopped.
[0144] Furthermore, the generating unit is also used for:
[0145] In response to the risk content description information being audio information or video information, preprocessing the audio information or video information to generate text information or picture information;
[0146] The generating unit is specifically used for:
[0147] The text information or picture information is subjected to multimodal content understanding.
[0148] Optionally, the generating unit is specifically used for:
[0149] Using a first text model to understand the text content of the text information, wherein the first text model is a large language model fine-tuned using historical risk data, and is used to extract target risk text from the text information;
[0150] A first image model is used to understand the image content of the image information, wherein the first image model is a multimodal model fine-tuned using historical risk data and is used to extract target risk images from the image information.
[0151] Optionally, the construction unit is specifically used for:
[0152] Using a second text model to understand the target risk text, and construct at least one risk content identification task, wherein the second text model is a large language model that uses historical risk data for supervised fine-tuning and preference alignment, and is used to search for a target risk text suitable for constructing a risk content identification task among multiple target risk texts;
[0153] A second image model is used to perform risk understanding on the target risk text and construct at least one risk content identification task, wherein the second image model is a multimodal model that uses historical risk data for supervised fine-tuning and preference alignment, and is used to search for a target risk image suitable for constructing a risk content identification task among multiple target risk images.
[0154] Optionally, the acquiring unit is specifically used for:
[0155] Build the task search process based on the LangGraph framework;
[0156] The at least one risk content identification task queries the content search system according to the task search process to obtain the N recalled contents.
[0157] Figure 5 Schematic diagram of the structure of the electronic device in the embodiment of the present invention. Figure 5 As shown, it includes a general computer hardware structure, which at least includes a processor 501 and a memory 502. The processor 501 and the memory 502 are connected via a bus 503. The memory 502 is suitable for storing instructions or programs executable by the processor 501. The processor 501 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 501 executes the instructions stored in the memory 502, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 503 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 504 and the display device and the input / output (I / O) device 505. The input / output (I / O) device 505 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 505 is connected to the system through an input / output (I / O) controller 506.
[0158] Among them, the instructions stored in the memory 502 are executed by at least one processor 501 to achieve: obtaining risk content description information; performing multimodal content understanding on the risk content description information to generate at least one target risk text or target risk picture; performing risk understanding on the at least one target risk text, or the target risk picture, or the risk content description information to construct at least one risk content identification task; querying the content search system according to the at least one risk content identification task to obtain N recalled contents, wherein the recalled contents are contents related to the risk content description information, the target risk text or the target risk picture, and N is a positive integer greater than or equal to 1; determining the discrimination result of each of the recalled contents.
[0159] Specifically, the electronic device includes: one or more processors 501 and a memory 502, Figure 5 Take a processor 501 as an example. The processor 501 and the memory 502 may be connected via a bus or other means. Figure 5 In the example, the bus connection is used. The memory 502 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 501 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 502, that is, the above-mentioned method for automatically identifying risk content is implemented.
[0160] The memory 502 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 502 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 502 may optionally include a memory remotely arranged relative to the processor 501, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0161] One or more modules are stored in the memory 502, and when executed by one or more processors 501, the method for automatically identifying risky content in any of the above method embodiments is executed.
[0162] As will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, various aspects of the embodiments of the present invention may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." In addition, various aspects of the embodiments of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer-readable program code implemented thereon.
[0163] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of an embodiment of the present invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.
[0164] A computer readable signal medium may include a propagated digital signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.
[0165] Program code embodied on a computer readable medium may be transmitted using any appropriate medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0166] The computer program code for performing operations for various aspects of the embodiments of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, etc., and conventional process programming languages such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer as a stand-alone software package; partially on the user's computer and partially on a remote computer; or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using the Internet of an Internet service provider).
[0167] The flowchart legend and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention described above describe various aspects of the embodiment of the present invention.It will be understood that each block of the flowchart legend and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instructions (executed by the processor of the computer or other programmable data processing device) create a device for implementing the function / action specified in the flowchart and / or block diagram block or block.
[0168] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing device, or other apparatus to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.
[0169] The computer program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus so that a series of operable steps are performed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or blocks.
[0170] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse. Users' refusal to process personal information other than the necessary information required for basic functions will not affect the user's use of basic functions.
Claims
1. A method for automatically identifying risky content, characterized in that: The method comprises: Obtain risk content description information; Performing multimodal content understanding on the risk content description information to generate at least one target risk text or target risk picture; Perform risk understanding on the at least one target risk text, the target risk picture, or the risk content description information, and construct at least one risk content identification task; querying a content search system according to the at least one risk content identification task to obtain N recalled contents, wherein the recalled contents are contents related to the risk content description information, the target risk text or the target risk picture, and N is a positive integer greater than or equal to 1; Determine the judgment result of each of the recalled contents.
2. The method according to claim 1, characterized in that The method further comprises: The determination result is output as at least one recall content that has a risk.
3. The method according to claim 1, characterized in that The method further comprises: In response to the judgment result that the N recalled contents have risks, the recalled contents with the judgment result that the risks exist are added to the risk content description information, and the multimodal content understanding, the risk understanding and subsequent processing processes are continued.
4. The method according to claim 1, characterized in that: The method further comprises: In response to the judgment result that there is no risk in the N recalled contents, the recalled contents with the judgment result that there is no risk are added to the risk content description information, and the multimodal content understanding, risk understanding and subsequent processing processes are continued, wherein the recalled contents that do not have risk are negative feedback.
5. The method according to claim 3 or 4, characterized in that: The method further comprises: In response to the number of times the recalled content is obtained being less than or equal to a set threshold, the next query of the content search system is stopped.
6. The method according to claim 1, characterized in that The method further includes: In response to the risk content description information being audio information or video information, preprocessing the audio information or video information to generate text information or picture information; The multimodal content understanding of the risk content description information specifically includes: The text information or picture information is subjected to multimodal content understanding.
7. The method according to claim 6, characterized in that The performing multimodal content understanding on the text information or the picture information specifically includes: Using a first text model to understand the text content of the text information, wherein the first text model is a large language model fine-tuned using historical risk data, and is used to extract target risk text from the text information; A first image model is used to understand the image content of the image information, wherein the first image model is a multimodal model fine-tuned using historical risk data and is used to extract target risk images from the image information.
8. The method according to claim 1, characterized in that The performing risk understanding on the at least one target risk text or the target risk picture to construct at least one risk content identification task specifically includes: Using a second text model to understand the target risk text, and construct at least one risk content identification task, wherein the second text model is a large language model that uses historical risk data for supervised fine-tuning and preference alignment, and is used to search for a target risk text suitable for constructing a risk content identification task among multiple target risk texts; A second image model is used to understand the risk of the target risk image and construct at least one risk content identification task, wherein the second image model is a multimodal model that uses historical risk data for supervised fine-tuning and preference alignment, and is used to search for a target risk image suitable for constructing a risk content identification task among multiple target risk images.
9. The method according to claim 1, characterized in that: The querying of the content search system according to the at least one risk content identification task to obtain N recalled contents specifically includes: Build the task search process based on the LangGraph framework; The at least one risk content identification task queries the content search system according to the task search process to obtain the N recalled contents.
10. A device for automatically identifying risky content, characterized in that: The device comprises: An acquisition unit, used to acquire risk content description information; A generating unit, configured to perform multimodal content understanding on the risk content description information to generate at least one target risk text or target risk picture; A construction unit, configured to perform risk understanding on the at least one target risk text, or the target risk picture, or the risk content description information, and construct at least one risk content identification task; The acquisition unit is further used to query the content search system according to the at least one risk content identification task to acquire N recalled contents, wherein the recalled contents are contents related to the target risk text or target risk picture, and N is a positive integer greater than or equal to 1; A determination unit is used to determine a determination result of each of the recalled contents.
11. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.