Detection method and device
Through server-side detection methods and devices, content detection is carried out on user sub-objects, and intention information is identified using large-scale language models, which solves the problem of errors and missed review of information content in the existing technology, and improves the accuracy of information security protection and content review .
Patent Information
- Application Number
- CN202410455446.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-16
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-04-16
AI Technical Summary
The existing machine review + human review methods are prone to errors and omissions in information content review, and it is especially difficult to identify "hints" that are not sensitive words but the content is in violation of regulations and "hints" in understanding specific fields, which leads to an impact on unsuitable groups.
Through the detection method and device on the server side, the detection request of the user terminal is received, the sub-objects of the first user and the second user are detected, intent information and security scores are obtained, and pre-processing results are generated, including interception, alarm and content update processing. , using large-scale language models for content security rating.
It realizes content detection of multi-user sub-objects, identifying the intention of third-party users, improves information security protection, reduces errors and missed reviews, and enhances the ability to identify and process unsuitable content.
Smart Images

Figure CN118245970B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of information security technology, and in particular to a detection method and device. Background Art
[0002] As computer hardware and data storage capabilities continue to improve, people are beginning to try to use complex neural network models to achieve more detailed and precise artificial intelligence tasks, such as natural language processing, image recognition, speech recognition, and so on.
[0003] In the above context, big model AI has quietly emerged. Big model AI refers to the technology that realizes AI by using large-scale neural network models. Large-scale neural network models usually require a large amount of data and computing resources for model training. Therefore, in the process of model training, it is usually necessary to use cloud computing, distributed computing and other technologies.
[0004] The advantage of large-model artificial intelligence technology is that it can simulate human intelligence to a certain extent, has strong generalization ability and robustness, and can demonstrate good performance in different tasks and scenarios.
[0005] At present, there are scenarios for applying large-scale artificial intelligence in various fields. In one example, when users process various information (including pictures, texts, videos, etc.), they often encounter various information security issues. For example, some information should not be viewed by young users, and some information is not suitable for users to use or view for a long time.
[0006] Existing information content review is implemented through a combination of machine review and human review. Machine review requires objective standards, but objective standards cannot resolve various "hints" (no sensitive words but illegal content). Human review may make mistakes and miss some content. It is also difficult to understand the corresponding "hints" in specific fields (illegal content is deduced from the context), which may affect inappropriate people. Summary of the invention
[0007] In view of this, an embodiment of the present invention provides a detection method and device to solve the problem that the existing machine review + human review method is prone to errors and omissions.
[0008] In a first aspect, an embodiment of the present invention provides a detection method, which is applied to a server and includes:
[0009] Receiving a detection request sent by a user terminal, the detection request including an object to be detected, the object to be detected including a first sub-object of a first user using the user terminal and a second sub-object of a second user as a third party;
[0010] Performing content detection on the first sub-object and the second sub-object respectively to obtain content detection results;
[0011] Sending a detection response to the user terminal, the detection response including the content detection result, so that the user terminal displays the content detection result to the first user;
[0012] The content detection result includes the intention information of the second user when publishing the second sub-object and the preprocessing result of the object to be detected.
[0013] Optionally, the first sub-object further includes a user identifier of the first user;
[0014] The performing content detection on the first sub-object and the second sub-object respectively to obtain content detection results specifically includes:
[0015] Acquire, according to the user identifier of the first user, a historical record corresponding to the first user;
[0016] extracting content features and behavior features from the first sub-object and the second sub-object respectively according to the object types to which the first sub-object and the second sub-object belong;
[0017] Determining generation scenarios of the first sub-object and the second sub-object according to the content feature and the behavior feature;
[0018] Under the generated scenario, the content feature and the behavior feature are respectively compared with the historical record and the scenario template corresponding to the generated scenario;
[0019] Determine, according to the comparison result, the intention information of the second user when publishing the second sub-object, and perform a security score for the object to be detected;
[0020] generating the preprocessing result according to the intention information of the second user when publishing the second sub-object and the security score;
[0021] The scenario template includes marked positive sample sub-data and negative sample sub-data; the pre-processing result includes one or more combinations of interception processing, alarm processing and content update processing for the second sub-object.
[0022] Optionally, before receiving the detection request sent by the user terminal, the method further includes:
[0023] Get sample data belonging to different object types;
[0024] Identify sample data of each object type, and mark positive sample sub-data and negative sample sub-data in the sample data;
[0025] Generate scene templates belonging to different object types using the positive sample sub-data and the negative sample sub-data;
[0026] The object type includes one or more combinations of text type, video type, voice type, and picture type.
[0027] Optionally, the method further comprises:
[0028] A user content detection entry is generated locally, wherein the user content detection entry includes a user identification field and a historical object record field, wherein the historical object record field stores detected objects and content detection results corresponding to the detected objects.
[0029] Optionally, the server includes a content security model;
[0030] After the positive sample sub-data and the negative sample sub-data are marked in the sample data, the method further includes:
[0031] Using the positive sample sub-data and the negative sample sub-data, an original large-scale language model LLM is trained to obtain the content security model;
[0032] The detection request is transmitted to the content security model, so that the content security model performs content detection on the object to be detected and outputs the content detection result.
[0033] In a second aspect, an embodiment of the present invention provides a detection device, which is applied to a server and includes:
[0034] A receiving unit, configured to receive a detection request sent by a user terminal, wherein the detection request includes an object to be detected, and the object to be detected includes a first sub-object of a first user using the user terminal and a second sub-object of a second user as a third party;
[0035] A detection unit, configured to perform content detection on the first sub-object and the second sub-object respectively to obtain a content detection result;
[0036] a sending unit, configured to send a detection response to the user terminal, the detection response including the content detection result, so that the user terminal displays the content detection result to the first user;
[0037] The content detection result includes the intention information of the second user when publishing the second sub-object and the preprocessing result of the object to be detected.
[0038] Optionally, the first sub-object further includes a user identifier of the first user;
[0039] The detection unit is specifically configured to obtain, according to the user identifier of the first user, a historical record corresponding to the first user;
[0040] extracting content features and behavior features from the first sub-object and the second sub-object respectively according to the object types to which the first sub-object and the second sub-object belong;
[0041] Determining generation scenarios of the first sub-object and the second sub-object according to the content feature and the behavior feature;
[0042] Under the generated scenario, the content feature and the behavior feature are respectively compared with the historical record and the scenario template corresponding to the generated scenario;
[0043] Determine, according to the comparison result, the intention information of the second user when publishing the second sub-object, and perform a security score for the object to be detected;
[0044] generating the preprocessing result according to the intention information of the second user when publishing the second sub-object and the security score;
[0045] The scenario template includes marked positive sample sub-data and negative sample sub-data; the pre-processing result includes one or more combinations of interception processing, alarm processing and content update processing for the second sub-object.
[0046] Optionally, the device further comprises:
[0047] An acquisition unit, used for acquiring sample data belonging to different object types;
[0048] An identification unit, used to identify sample data of each object type and mark positive sample sub-data and negative sample sub-data in the sample data;
[0049] A first generating unit, configured to generate scene templates belonging to different object types by using the positive sample sub-data and the negative sample sub-data;
[0050] The object type includes one or more combinations of text type, video type, voice type, and picture type.
[0051] Optionally, the device further comprises:
[0052] The second generating unit is used to generate a user content detection table entry locally, wherein the user content detection table entry includes a user identification field and a historical object record field, and the historical object record field stores a detected object and a content detection result corresponding to the detected object.
[0053] Optionally, the server includes a content security model;
[0054] The device further includes: a training unit, configured to train an original large-scale language model LLM using the positive sample sub-data and the negative sample sub-data to obtain the content security model;
[0055] A transmission unit is used to transmit the detection request to the content security model, so that the content security model performs content detection on the object to be detected and outputs the content detection result.
[0056] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside a space enclosed by the housing, and the processor and the memory are arranged on the circuit board; a power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the detection method described in the method of the first aspect above.
[0057] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the detection method described in the first aspect above.
[0058] An embodiment of the present invention provides a detection method and device, in which a server receives a detection request sent by a user terminal, the detection request includes an object to be detected, and the object to be detected includes a first sub-object of a first user using the user terminal and a second sub-object of a second user who is a third party; the server performs content detection on the first sub-object and the second sub-object respectively to obtain a content detection result; the server sends a detection response to the user terminal, the detection response includes the content detection result, so that the user terminal displays the content detection result to the first user; wherein the content detection result includes the intention information of the second user when publishing the second sub-object and the preprocessing result of the object to be detected.
[0059] The above solution detects the content of sub-objects of multiple users to obtain the intention information of third-party users and the pre-processing results of sub-objects. It realizes the recognition of the intention of third-party users through large-model artificial intelligence technology, and then provides security protection for the current users. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0061] Figure 1 A flow chart of a detection method provided by an embodiment of the present invention;
[0062] Figure 2 A schematic diagram of the structure of a detection device provided in an embodiment of the present invention;
[0063] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0064] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0065] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] The following is combined with Figure 1 , the scheme provided by the embodiment of the present invention is described in detail, Figure 1 The flowchart of a detection method provided by an embodiment of the present invention is a flow chart. In the embodiment of the present invention, the implementation subject is an electronic device. The electronic device may be a server. Figure 1 As shown, the method of this embodiment specifically includes the following steps:
[0067] Step 110: receiving a detection request sent by a user terminal, wherein the detection request includes an object to be detected, and the object to be detected includes a first sub-object of a first user using the user terminal and a second sub-object of a second user who is a third party;
[0068] In this step, the first user uses the user terminal to browse content, and the above content browsing includes but is not limited to reading text, viewing pictures, viewing short videos, chatting, etc. Among them, the text, pictures, and short videos can be published on the Internet by the second user as a third party; the chat conversation can be a real-time communication process between the first user and the second user.
[0069] When the first user is browsing content, the user terminal collects the content browsed by the first user and generates a detection request, which includes an object to be detected. The object to be detected includes a first sub-object of the first user and a second sub-object of the second user.
[0070] In one example, if the first user is having a chat conversation with the second user, the object to be detected includes the entire content of the chat conversation between the first user and the second user. The first sub-object includes the chat content input by the first user; the second sub-object includes the chat content input by the second user. The chat content includes but is not limited to text, pictures, videos, links, emoticons, voice, etc.
[0071] In another example, if the first user is watching a short video, the object to be detected includes the entire process of the first user selecting, confirming, and watching the short video. The first sub-object includes the selection instruction, confirmation instruction, etc. input by the first user; the second sub-object includes the short video posted by the second user, the short video content introduction, etc.
[0072] The user terminal sends a detection request to the server to request the server to perform a security detection on the object to be detected.
[0073] Step 120: Perform content detection on the first sub-object and the second sub-object respectively to obtain content detection results;
[0074] In this step, after receiving the detection request, the server obtains the object to be detected from it.
[0075] Optionally, the first sub-object further includes a user identifier of the first user, which can uniquely represent the first user, for example, the first user's name, online name, WeChat name, etc. Through the user identifier of the first user, the server can identify the first sub-object and the second sub-object, that is, the sub-object with the user identifier is the first sub-object, and the sub-object without the user identifier is the second sub-object.
[0076] It is understandable that, in actual applications, the user ID of each user may be obtained, and the user ID may be marked for each user's sub-object, so that the server can quickly divide the sub-objects.
[0077] The server performs content detection on each sub-object respectively to obtain a content detection result.
[0078] Optionally, the server performs content detection on each sub-object respectively to obtain a content detection result, specifically:
[0079] According to the user identifier of the first user, the server first obtains the historical record corresponding to the first user; according to the object type to which the first sub-object and the second sub-object belong, the server extracts content features and behavior features from the first sub-object and the second sub-object respectively; according to the content features and the behavior features, the server determines the generation scene of the first sub-object and the second sub-object; under the generation scene, the server compares the content features and the behavior features with the historical records and the scene template corresponding to the generation scene respectively; according to the comparison result, the server determines the intention information of the second user when publishing the second sub-object, and performs a security score for the object to be detected; according to the intention information and the security score when the second user publishes the second sub-object, the server generates a preprocessing result.
[0080] The scene template includes the marked positive sample sub-data and negative sample sub-data; the pre-processing result includes one or more combinations of interception processing, alarm processing and content update processing for the second sub-object.
[0081] The above-mentioned content features may be specifically text features; the behavior features may be specifically action features. The above-mentioned generated scene may be specifically the scene where the first user and the second user are currently located, which is identified by the server based on the first sub-object and the second sub-object. For example, a dating and chatting scene, a video watching scene, a text browsing and picture browsing scene, a fraud scene, a pig-killing scene, and so on. The above-mentioned scene template is specifically a standardized format corresponding to each scene, and the positive sample sub-data and the negative sample sub-data are respectively marked in the scene template, so that in the comparison process, the location where the negative sample sub-data appears can be quickly determined, and the intent information can be determined. The above-mentioned intent information may specifically include the intent information of the first user and the intent information of the second user. For example, the intent information of the first user entering a sentence, the intent information of watching a short video, the intent information of the second user entering a sentence, the intent information of publishing a short video, and so on.
[0082] In one example, a first user uses a user terminal to have a chat conversation with a second user, and the user terminal uses the entire chat conversation content as an object to be detected and sends it to a server via a detection request. After receiving the detection request, the server obtains the object to be detected from it and distinguishes the first sub-object and the second sub-object according to the user identifier.
[0083] The server obtains the history records of the first user according to the user ID of the first user. The history records include the chat contents (chat contents with parents and friends, language style, dialogue mode, etc.) and chat behaviors (input habits, time, location, etc.) of the first user with other users. The server recognizes that the object types to which the first sub-object and the second sub-object belong are both text types.
[0084] According to the text type, the server extracts content features and behavior features from the first sub-object and the second sub-object respectively. For example, the text content input by each user (life and work information, personal identity information, personal account, password, etc. input by the first user; life and work information input by the second user, asking the identity information of the first user, asking the personal account, password, etc. of the first user), the language style of each user when inputting text (for example, casual, formal, etc.), the input habits of each user (for example, pinyin input, voice input, handwriting input, etc.), and the time of each user's chat (for example, usually between 8 and 9 pm, etc.).
[0085] After the server obtains the above content features and behavior features, it uses the content features and behavior features to determine the generation scene. In this example, the server determines that the first user and the second user are in a chat dialogue scene through the content features and behavior features.
[0086] In the chat dialogue scenario, the server compares the above content features and behavior features with the historical records and the chat dialogue scenario template corresponding to the chat dialogue scenario. For example, the content features and behavior features of the first user are compared with the historical records of the first user. The content features and text features of the first user and the second user are compared with the chat dialogue scenario template.
[0087] It is understandable that the chat dialogue scenario template includes positive sample sub-data and negative sample sub-data, and the attributes of the sub-data are marked. Among them, the positive sample sub-data is the content of the normal chat dialogue between users, and the negative sample sub-data is the content of the abnormal chat dialogue between users. For example, in the process of the chat dialogue between the first user and the second user, the initial stage is the content of the normal chat dialogue, the content of the mutual greeting, the inquiry of the details of life, the text input behavior, etc., in the subsequent stage, the content of the abnormal chat dialogue appears, such as asking for money, asking for account numbers, passwords, and voice input. At this time, the server judges that the content of the mutual greeting, the inquiry of the details of life, and the text input behavior are all the content of the normal chat dialogue based on the historical records of the first user and the chat dialogue scenario template, and the asking for money, asking for account numbers, passwords, and voice input match the negative sample sub-data in the chat dialogue scenario template, and the server judges that these are the content of the abnormal chat dialogue.
[0088] After obtaining the comparison result, the server determines that the second user's intention is to induce the first user to transfer property. The server performs a security score on the object to be detected. For example, in this example, the security score is 1.
[0089] Based on the intention of inducing the first user to transfer property and the security score of 1, the server generates a preprocessing result of intercepting the first user from transferring property and prompting the user that there is a risk of property loss.
[0090] It should be noted that the server is configured with a security scoring strategy. The security scoring strategy includes a security score, which is specifically 1 to 10 points. The higher the score, the higher the security level. Each score in the security score matches the intent information. For example, if the intent information indicates inducing the transfer of property or fraud, the security score is 1; if the intent information indicates a tendency to violence or blood, the security score is 2, and so on.
[0091] The above content update process specifically refers to the server blocking, deleting, and rewriting content with security risks. For example, based on historical records, when the server determines that the first user is a teenager, it blocks, deletes, and rewrites violent and bloody content; when the server determines that the second user's intention is to induce the first user to transfer property, it blocks, deletes, and rewrites the content that represents the second user's intention.
[0092] Step 130: Send a detection response to the user terminal, where the detection response includes the content detection result, so that the user terminal displays the content detection result to the first user.
[0093] In this step, after obtaining the content detection result, the server generates a detection response, which includes the content detection result.
[0094] The server sends a detection response to the user terminal. After receiving the detection response, the user terminal obtains the content detection result therefrom and displays the content detection result to the first user.
[0095] The first user can decide whether to continue the chat conversation with the second user based on the content detection result.
[0096] By applying the detection method provided by an embodiment of the present invention, a server receives a detection request sent by a user terminal, the detection request includes an object to be detected, and the object to be detected includes a first sub-object of a first user using the user terminal and a second sub-object of a second user as a third party; the server performs content detection on the first sub-object and the second sub-object respectively to obtain a content detection result; the server sends a detection response to the user terminal, the detection response includes the content detection result, so that the user terminal displays the content detection result to the first user; wherein the content detection result includes the intention information of the second user when publishing the second sub-object and the preprocessing result of the object to be detected.
[0097] The above solution detects the content of sub-objects of multiple users to obtain the intention information of third-party users and the pre-processing results of sub-objects. It realizes the recognition of the intention of third-party users through large-model artificial intelligence technology, and then provides security protection for the current users.
[0098] Optionally, before the present invention implements step 110, the server will also perform a process of marking positive and negative sample sub-data according to sample data of different object types.
[0099] Specifically, the administrator imports sample data belonging to different object types into the server. After the server obtains the sample data, it identifies the sample data of each object type and marks the positive sample sub-data (friendship chat data, love data) and negative sample sub-data in the sample data. Among them, the negative sample sub-data is data with security risks (for example, pig-killing data, fraud data, etc.).
[0100] Using the positive sample sub-data and the negative sample sub-data, the server generates scene templates belonging to different object types.
[0101] The object type includes one or more combinations of text type, video type, voice type, and picture type.
[0102] Optionally, in the embodiment of the invention, the process of the server storing historical object records locally is also included.
[0103] Specifically, the server generates a user content detection entry locally, and the user content detection entry includes a user identification field and a historical object record field, and the historical object record field stores detected objects and content detection results corresponding to the detected objects.
[0104] Optionally, in the embodiment of the present invention, the server may further include a content security model, and the content security model may implement security detection of the object to be detected.
[0105] Specifically, after the server marks the positive sample sub-data and the negative sample sub-data in the sample data, the server inputs the positive sample sub-data and the negative sample sub-data into the original large-scale language model, and trains the LLM to obtain a content security model.
[0106] Subsequently, after receiving the detection request, the server transmits the detection request to the content security model, so that the content security model performs content detection on the object to be detected and outputs the content detection result.
[0107] In the embodiment of the present invention, the process of performing content detection on the object to be detected by the content security model is the same as the process of the aforementioned step 120, which will not be repeated here.
[0108] Figure 3 A schematic diagram of the structure of a detection device provided in an embodiment of the present invention is shown in FIG. Figure 3 As shown, the detection device of this embodiment is applied to a server, and the device may include: a receiving unit 310 , a detection unit 320 and a sending unit 330 .
[0109] The receiving unit 310 is configured to receive a detection request sent by a user terminal, wherein the detection request includes an object to be detected, and the object to be detected includes a first sub-object of a first user using the user terminal and a second sub-object of a second user as a third party;
[0110] A detection unit 320, configured to perform content detection on the first sub-object and the second sub-object respectively to obtain a content detection result;
[0111] A sending unit 330 is configured to send a detection response to the user terminal, where the detection response includes the content detection result, so that the user terminal displays the content detection result to the first user;
[0112] The content detection result includes the intention information of the second user when publishing the second sub-object and the preprocessing result of the object to be detected.
[0113] Optionally, the first sub-object further includes a user identifier of the first user;
[0114] The detection unit 320 is specifically configured to obtain a history record corresponding to the first user according to the user identifier of the first user;
[0115] extracting content features and behavior features from the first sub-object and the second sub-object respectively according to the object types to which the first sub-object and the second sub-object belong;
[0116] Determining generation scenarios of the first sub-object and the second sub-object according to the content feature and the behavior feature;
[0117] Under the generated scenario, the content feature and the behavior feature are respectively compared with the historical record and the scenario template corresponding to the generated scenario;
[0118] Determine, according to the comparison result, the intention information of the second user when publishing the second sub-object, and perform a security score for the object to be detected;
[0119] generating the preprocessing result according to the intention information of the second user when publishing the second sub-object and the security score;
[0120] The scenario template includes marked positive sample sub-data and negative sample sub-data; the pre-processing result includes one or more combinations of interception processing, alarm processing and content update processing for the second sub-object.
[0121] Optionally, the device further comprises:
[0122] An acquisition unit (not shown in the figure), used to acquire sample data belonging to different object types;
[0123] An identification unit (not shown in the figure), used to identify sample data of each object type and mark positive sample sub-data and negative sample sub-data in the sample data;
[0124] A first generating unit (not shown in the figure), configured to generate scene templates belonging to different object types using the positive sample sub-data and the negative sample sub-data;
[0125] The object type includes one or more combinations of text type, video type, voice type, and picture type.
[0126] Optionally, the device further comprises:
[0127] The second generating unit (not shown in the figure) is used to locally generate a user content detection table entry, wherein the user content detection table entry includes a user identification field and a historical object record field, wherein the historical object record field stores detected objects and content detection results corresponding to the detected objects.
[0128] Optionally, the server includes a content security model;
[0129] The device further includes: a training unit (not shown in the figure), configured to train an original large-scale language model LLM using the positive sample sub-data and the negative sample sub-data to obtain the content security model;
[0130] A transmission unit (not shown in the figure) is used to transmit the detection request to the content security model, so that the content security model performs content detection on the object to be detected and outputs the content detection result.
[0131] The device of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.
[0132] Accordingly, the detection device provided in the embodiment of the present invention may also be implemented with another structure. Figure 3 A schematic diagram of the structure of an electronic device embodiment provided by the present invention can implement the present invention Figure 1 The process of the embodiment shown is as follows: Figure 3As shown, the electronic device may include: a housing 41, a processor 42, a memory 43, a circuit board 44 and a power supply circuit 45, wherein the circuit board 44 is placed inside the space enclosed by the housing 41, and the processor 42 and the memory 43 are arranged on the circuit board 44; the power supply circuit 45 is used to supply power to various circuits or devices of the electronic device; the memory 43 is used to store executable program codes; the processor 42 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 43, so as to execute the method described in the above-mentioned embodiment.
[0133] For details on the specific execution process of the above steps by the processor 42 and the steps further executed by the processor 42 by running the executable program code, please refer to the present invention. Figure 1 The description of the illustrated embodiment will not be repeated here.
[0134] The electronic device is a device that provides computing services. The electronic device consists of a processor, hard disk, memory, system bus, etc. The electronic device is similar to the general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, manageability, etc.
[0135] It should be noted that, in this article, terms such as "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0136] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0137] In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0138] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or in conjunction with such instruction execution system, apparatus or device. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, apparatus or device, or in conjunction with such instruction execution system, apparatus or device.
[0139] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0140] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof.
[0141] In the above embodiments, multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it may be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0142] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0143] For the convenience of description, the above device is described by dividing the functions into various units / modules. Of course, when implementing the present invention, the functions of each unit / module can be implemented in the same or multiple software and / or hardware.
[0144] It can be known from the above description of the implementation mode that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.
Claims
1. A detection method, characterized in that: The method is applied to a server, and the method comprises: Receiving a detection request sent by a user terminal, the detection request including an object to be detected, the object to be detected including a first sub-object of a first user using the user terminal and a second sub-object of a second user as a third party; the receiving of the detection request sent by the user terminal includes: during the content browsing process of the first user using the user terminal, the user terminal collects the content browsed by the first user and generates a detection request and sends it to a server; the content browsing includes: reading text, viewing pictures and viewing short videos; Performing content detection on the first sub-object and the second sub-object respectively to obtain content detection results; Sending a detection response to the user terminal, the detection response including the content detection result, so that the user terminal displays the content detection result to the first user; The content detection result includes the intention information of the second user when publishing the second sub-object and the pre-processing result of the object to be detected; The first sub-object also includes a user identifier of the first user; The performing content detection on the first sub-object and the second sub-object respectively to obtain content detection results specifically includes: Acquire, according to the user identifier of the first user, a history record corresponding to the first user; the history record includes chat content and chat behavior of the first user with other users in the past; extracting content features and behavior features from the first sub-object and the second sub-object respectively according to the object types to which the first sub-object and the second sub-object belong; Determining generation scenarios of the first sub-object and the second sub-object according to the content feature and the behavior feature; Under the generated scenario, the content feature and the behavior feature are respectively compared with the historical record and the scenario template corresponding to the generated scenario; Determine, based on the comparison result, the intention information of the second user when publishing the second sub-object, and perform a security score for the object to be detected; generating the preprocessing result according to the intention information of the second user when publishing the second sub-object and the security score; The scenario template includes the positive sample sub-data and the negative sample sub-data marked; the pre-processing result includes one or more combinations of interception processing, alarm processing and content update processing for the second sub-object; The server is configured with a security scoring policy, wherein the security scoring policy includes a security score, wherein each score in the security score matches the intention information.
2. The method according to claim 1, characterized in that: Before receiving the detection request sent by the user terminal, the method further includes: Get sample data belonging to different object types; Identify sample data of each object type, and mark positive sample sub-data and negative sample sub-data in the sample data; Generate scene templates belonging to different object types using the positive sample sub-data and the negative sample sub-data; The object type includes one or more combinations of text type, video type, voice type, and picture type.
3. The method according to claim 1, characterized in that The method further comprises: A user content detection entry is generated locally, wherein the user content detection entry includes a user identification field and a historical object record field, wherein the historical object record field stores detected objects and content detection results corresponding to the detected objects.
4. The method according to claim 2, characterized in that: The server includes a content security model; After the positive sample sub-data and the negative sample sub-data are marked in the sample data, the method further includes: Using the positive sample sub-data and the negative sample sub-data, an original large-scale language model LLM is trained to obtain the content security model; The detection request is transmitted to the content security model, so that the content security model performs content detection on the object to be detected and outputs the content detection result.
5. A detection device, characterized in that: The device is applied to a server, and comprises: A receiving unit, configured to receive a detection request sent by a user terminal, wherein the detection request includes an object to be detected, and the object to be detected includes a first sub-object of a first user using the user terminal and a second sub-object of a second user as a third party; The receiving unit is specifically configured to collect the content browsed by the first user using the user terminal and generate a detection request to send to the server during the content browsing process of the first user using the user terminal; the content browsing includes: reading text, viewing pictures and viewing short videos; A detection unit, configured to perform content detection on the first sub-object and the second sub-object respectively to obtain a content detection result; a sending unit, configured to send a detection response to the user terminal, the detection response including the content detection result, so that the user terminal displays the content detection result to the first user; The content detection result includes the intention information of the second user when publishing the second sub-object and the pre-processing result of the object to be detected; The first sub-object also includes a user identifier of the first user; The detection unit is specifically configured to obtain, according to the user identifier of the first user, a history record corresponding to the first user; the history record includes chat content and chat behavior of the first user with other users in the past; extracting content features and behavior features from the first sub-object and the second sub-object respectively according to the object types to which the first sub-object and the second sub-object belong; Determining generation scenarios of the first sub-object and the second sub-object according to the content feature and the behavior feature; Under the generated scenario, the content feature and the behavior feature are respectively compared with the historical record and the scenario template corresponding to the generated scenario; Determine, based on the comparison result, the intention information of the second user when publishing the second sub-object, and perform a security score for the object to be detected; generating the preprocessing result according to the intention information of the second user when publishing the second sub-object and the security score; The scenario template includes marked positive sample sub-data and negative sample sub-data; the preprocessing result includes one or more combinations of interception processing, alarm processing and content update processing for the second sub-object; a security scoring strategy is configured in the server, and the security scoring strategy includes a security score, wherein each score in the security score matches the intent information.
6. The device according to claim 5, characterized in that The device also includes: An acquisition unit, used for acquiring sample data belonging to different object types; An identification unit, used to identify sample data of each object type and mark positive sample sub-data and negative sample sub-data in the sample data; A first generating unit, configured to generate scene templates belonging to different object types by using the positive sample sub-data and the negative sample sub-data; The object type includes one or more combinations of text type, video type, voice type, and picture type.
7. The device according to claim 5, characterized in that The device also includes: The second generating unit is used to generate a user content detection table entry locally, wherein the user content detection table entry includes a user identification field and a historical object record field, and the historical object record field stores a detected object and a content detection result corresponding to the detected object.
8. The device according to claim 6, characterized in that The server includes a content security model; The device further includes: a training unit, configured to train an original large-scale language model LLM using the positive sample sub-data and the negative sample sub-data to obtain the content security model; A transmission unit is used to transmit the detection request to the content security model, so that the content security model performs content detection on the object to be detected and outputs the content detection result.
9. An electronic device, characterized in that: The electronic device comprises: a housing, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is placed inside the space enclosed by the housing, and the processor and the memory are arranged on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program codes; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the detection method described in any one of the preceding claims 1-4.
10. A computer-readable storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the detection method according to any one of claims 1 to 4.
