Content monitoring method and device and electronic equipment

Abstract

The invention provides a content monitoring method and device and electronic equipment, and relates to the technical field of information processing. The method comprises the following steps: receiving a to-be-detected content set from a client, wherein the to-be-detected content set comprises to-be-detected content of at least one mode; calling a modal recognition module in the monitoring model to recognize the modal of the to-be-detected content; after the target mode of the to-be-detected content is identified, a detection module used for detecting the content of the target mode in the monitoring model is called to detect whether the to-be-detected content meets a preset alarm condition or not; and if the to-be-detected content of any mode in the to-be-detected content set meets the alarm condition, sending alarm information to the client. According to the embodiment of the invention, not only can the content with relatively abundant forms be monitored and the monitoring performance be improved, but also the monitoring model is obtained based on deep learning and is beneficial to flexibly adapting to a scene with an updated content presentation form.

Description

Technical Field

[0001] The present invention belongs to the field of information processing technology, and specifically relates to a content monitoring method, device and electronic equipment. Background Art

[0002] With the development of Internet technology and the widespread use of smart devices, all kinds of information are flooding our daily lives, greatly increasing the risk of minors being exposed to inappropriate content. Based on this, it is necessary to monitor online content and isolate inappropriate content from minors.

[0003] At present, the mainstream content monitoring methods include identifying and blocking inappropriate content through banned word filtering, or managing the usage time and access rights of smart devices to block inappropriate content. However, these methods have poor flexibility and are difficult to effectively monitor content with updated presentation forms, resulting in high false alarm and missed alarm rates. In addition, the mainstream content monitoring methods can only cover a relatively single content format. For example, banned words can only effectively monitor text-based network content, and it is difficult to effectively monitor image or audio-based network content. It can be seen that the current mainstream content monitoring methods have a single monitored content format, poor monitoring performance, and cannot flexibly adapt to scenarios with updated content presentation forms. Summary of the invention

[0004] The present disclosure proposes a content monitoring method, device and electronic device, which detect multiple modalities of content to be detected through a preset monitoring model, and output information according to whether the multiple modalities of content to be detected meet preset alarm conditions. It can not only monitor content with richer forms, which is beneficial to improving monitoring performance, but also the multiple algorithm modules included in the monitoring model are obtained based on deep learning, which is beneficial to flexible adaptation to scenarios with updated content presentation forms.

[0005] A first aspect of the present disclosure provides a content monitoring method, which is applied to a server. The method includes:

[0006] Receive a set of content to be detected from a client, wherein the set of content to be detected includes content to be detected in at least one mode, and the content to be detected in any mode is text, image or audio; the set of content to be detected is collected in response to a user operation after the client enters a monitoring mode;

[0007] For any content to be detected in the set of content to be detected, calling a modality identification module in the monitoring model to identify the modality of the content to be detected;

[0008] After identifying the target mode of the content to be detected, calling the detection module in the monitoring model for detecting the content of the target mode to detect whether the content to be detected meets the preset alarm condition;

[0009] If any mode of the to-be-detected content in the to-be-detected content set meets the alarm condition, an alarm message is sent to the client.

[0010] In the embodiment of the present disclosure, the method specifically detects whether the content to be detected meets the preset alarm condition in the following manner:

[0011] If the content to be detected is text, extracting semantic features and semantic representation words of the content to be detected, and using the semantic features and the semantic representation words as representation information;

[0012] If the content to be detected is an image, extracting image semantic features of the content to be detected, and using the image semantic features as representation information;

[0013] If the content to be detected is audio, extracting text semantic features of the text corresponding to the content to be detected and emotional features of the content to be detected, and using the text semantic features and the emotional features as representation information;

[0014] Determine whether the content to be detected meets the alarm condition according to the characterization information.

[0015] In an embodiment of the present disclosure, if the content to be detected is text, the method further includes:

[0016] If it is determined that the semantic representation word is a word that meets the alarm condition, detecting whether the preset violation word library contains the semantic representation word;

[0017] If the preset violation word library does not include the semantic representation word, the semantic representation word is added to the preset violation word library.

[0018] In the embodiment of the present disclosure, it also includes:

[0019] Determine the total number of received content sets to be detected;

[0020] If the total number is greater than or equal to the preset number, predicting the user intention according to the received set of content to be detected;

[0021] Determining whether the user's intention is potentially risky;

[0022] If the user's intention involves potential risks, a risk reminder message is sent to the client.

[0023] In the embodiment of the present disclosure, predicting the user intention according to the received content set to be detected includes:

[0024] For any modality, obtaining representation information of the content to be detected of the modality in each content set to be detected of the received content set to be detected;

[0025] Extracting latent features of the characterization information of the content to be detected in the modality based on the receiving timing of each of the received content sets to be detected, wherein the latent features are used to characterize changes of the content to be detected in the modality according to the receiving timing;

[0026] Predicting the user's intention towards the content to be detected in the modality according to the latent features.

[0027] In the embodiment of the present disclosure, it also includes:

[0028] Adding the to-be-detected content set and the detection results of each to-be-detected content in the to-be-detected content set to a preset training sample library to obtain an updated training sample library; wherein the to-be-detected content set is used as a training sample, and the detection results are used as labels of the training sample;

[0029] The parameters of the monitoring model are adjusted based on the updated training sample library to obtain an updated monitoring model.

[0030] An embodiment of a second aspect of the present disclosure provides a content monitoring method, which is applied to a client, and the method includes:

[0031] In the case of entering the monitoring mode, in response to the user's operation, collecting the to-be-detected content set of the client, wherein the to-be-detected content set includes to-be-detected content of at least one modality, and the to-be-detected content of any modality is text, image or audio;

[0032] The set of content to be detected is sent to the server, so that the server calls the modality identification module in the monitoring model to identify the modality of any content to be detected in the set of content to be detected; and after identifying the target modality of the content to be detected, calls the detection module in the monitoring model for detecting the content of the target modality to detect whether the content to be detected meets the preset alarm condition;

[0033] When the content to be detected in any mode in the set of content to be detected meets the alarm condition, receiving alarm information from the server.

[0034] An embodiment of a third aspect of the present disclosure provides a content monitoring device, which is applied to a server, and the device includes:

[0035] A receiving unit, configured to receive a set of content to be detected from a client, wherein the set of content to be detected includes content to be detected in at least one mode, and the content to be detected in any mode is text, image or audio; the set of content to be detected is collected in response to a user's operation after the client enters a monitoring mode;

[0036] an identification unit, configured to call a modality identification module in the monitoring model to identify a modality of any content to be detected in the set of content to be detected;

[0037] A detection unit, configured to, after identifying the target mode of the content to be detected, call a detection module in the monitoring model for detecting the content of the target mode to detect whether the content to be detected meets a preset alarm condition;

[0038] The sending unit is used to send an alarm message to the client if any mode of the to-be-detected content in the to-be-detected content set meets the alarm condition.

[0039] An embodiment of a fourth aspect of the present disclosure provides a content monitoring device, which is applied to a client, and the device includes:

[0040] a collection unit, configured to collect a set of to-be-detected content of the client in response to a user operation when entering the monitoring mode, wherein the set of to-be-detected content includes to-be-detected content of at least one modality, and the to-be-detected content of any modality is text, image or audio;

[0041] A sending unit, configured to send the set of content to be detected to a server, so that the server calls the modality identification module in the monitoring model to identify the modality of any content to be detected in the set of content to be detected; and after identifying the target modality of the content to be detected, calls the detection module in the monitoring model for detecting the content of the target modality to detect whether the content to be detected meets a preset alarm condition;

[0042] The receiving unit is used to receive the alarm information from the server when the content to be detected in any mode in the content to be detected set meets the alarm condition.

[0043] An embodiment of the fifth aspect of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect or the second aspect.

[0044] An embodiment of the sixth aspect of the present disclosure provides a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor to implement the method described in the first aspect or the second aspect above.

[0045] The technical solution provided in the embodiments of the present disclosure has at least the following technical effects or advantages:

[0046] The disclosed embodiment pre-sets a monitoring model for detecting client content, and the monitoring model may include a modality recognition module and a plurality of detection modules. The modality recognition module and the plurality of detection modules may be obtained based on deep learning, so that they are not restricted by fixed content and can flexibly adapt to the scenario of updated content presentation form through semantic understanding. Further, after the client enters the monitoring mode, in response to the user trigger, the content to be detected of at least one modality can be collected. Correspondingly, the server can call the monitoring model to identify the modality of the content to be detected of each modality, and detect whether the content to be detected meets the preset alarm condition. When the content to be detected of any modality meets the alarm condition, an alarm message is sent to the client. Wherein, the content to be detected of any modality is text, image or audio. It can be seen that the disclosed embodiment can monitor content with richer forms, which is conducive to improving monitoring performance.

[0047] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will become apparent from the following description or be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present disclosure. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0049] In the attached picture:

[0050] Figure 1 A structural diagram of a content monitoring system provided by an embodiment of the present disclosure is shown;

[0051] Figure 2 A schematic diagram of the structure of a monitoring model provided by an embodiment of the present disclosure is shown;

[0052] Figure 3 A method flow diagram of a content monitoring method provided by an embodiment of the present disclosure is shown;

[0053] Figure 4 A schematic diagram showing a method flow of another content monitoring method provided by an embodiment of the present disclosure is shown;

[0054] Figure 5A A schematic diagram showing the structure of a content monitoring device provided by an embodiment of the present disclosure is shown;

[0055] Figure 5B A schematic diagram showing the structure of a content monitoring device provided by another embodiment of the present disclosure is shown;

[0056] Figure 6A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown;

[0057] Figure 7 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0058] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present disclosure should have the common meanings understood by those skilled in the art to which the present disclosure belongs.

[0060] The following introduces the implementation scenarios and related technologies involved in the embodiments of the present disclosure.

[0061] The disclosed embodiments are intended to monitor network content that has an inappropriate impact on minors, so as to intervene in the corresponding content in a timely manner. With the development of Internet technology and the widespread use of smart devices, minors can access a variety of network content without obstacles, and there are many network contents that may have an inappropriate impact on minors or even lead to bad things, such as violent content, fraudulent content, etc. Therefore, when minors use smart devices, it is necessary to monitor network content and isolate inappropriate content from minors.

[0062] At present, a commonly used network content monitoring method includes identifying and blocking inappropriate content by scanning whether the words in the text contain preset keywords to intercept text content containing illegal words. In this method, an illegal word library can be pre-set, and the illegal word library can contain illegal words or phrases. Obviously, this method is only applicable to the detection and monitoring of text content, and cannot monitor image content and audio content. The content that can be covered is single in form. In addition, when the text expression changes or new network terms appear, it is difficult to update the illegal word library in real time, resulting in difficulty in detecting illegal content and causing omissions or misreporting. Another commonly used network content monitoring method includes using control software to manage the access rights of smart devices when used by minors to ensure that minors will not be exposed to inappropriate content. The monitoring principle of this type of control software is to predefine a website blacklist mode within a certain access time period, so that the smart device cannot access the websites included in the blacklist during the time period, thereby reducing the possibility of minors being exposed to inappropriate content. However, the control software restricts from the access level, cannot analyze network content in real time, and is difficult to detect the quality of network content actually exposed to minors. In addition, the predefined blacklist is relatively fixed, and it is not possible to flexibly and effectively monitor emerging network content.

[0063] In view of this, the technical solution proposed in the embodiment of the present disclosure pre-sets a monitoring model. Since the monitoring model can be obtained based on deep learning, it is not limited by fixed content and can flexibly adapt to the scene of updated content presentation form through semantic understanding. In addition, through the pre-set monitoring model, the content to be detected in multiple modes is detected, and information is output according to whether the content to be detected in multiple modes meets the preset alarm conditions, so that it is possible to monitor content with richer forms, which is conducive to improving monitoring performance.

[0064] The following describes a content monitoring method, system, device, electronic device and storage medium proposed according to an embodiment of the present disclosure. The technical solution of the present disclosure is described in detail with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present disclosure will be described below in conjunction with the accompanying drawings.

[0065] See also Figure 1 , Figure 1 An exemplary system structure diagram of a content monitoring system provided by the present disclosure is shown. The content monitoring system provided by the present disclosure includes a server 10 and a client 20. The server 10 and the client 20 communicate with each other through a network.

[0066] It is understandable that the structure shown in the present disclosure does not constitute a specific limitation on the content monitoring system. In other embodiments of the present disclosure, the content monitoring system may include more or less devices than shown in the figure, which is not limited here.

[0067] The server 10 can be an independent physical server, or a server cluster, a distributed system or a cloud platform composed of multiple physical servers. The server 10 can deploy resources and computing power for network content detection. Exemplarily, the server 10 can be used to store monitoring models, computing resources and data. The server 10 executes the various embodiments of the content monitoring method of the present disclosure by calling the monitoring model. Computing resources may include, for example, a vocabulary library (such as a violation vocabulary library) that supports the implementation of the technical solution of the present disclosure, a training sample library, and operating logic that supports the monitoring model (such as the logic of detecting content modality, the logic of fusion of detection results, the optimization logic of the monitoring model, etc.). The data is used to provide support for the server 10 to call the monitoring model.

[0068] The server 10 can be used to receive the content to be detected collected by the client 20, and then call the preset monitoring model to detect whether the content to be detected meets the preset alarm condition. The server 10 can also be used to receive the content to be detected collected by the client 20 multiple times, and based on the content to be detected received multiple times, call the monitoring model to predict the user's intention. Among them, the content to be detected received each time can be generated by the client 20 in response to the user trigger.

[0069] The alarm conditions described in the embodiments of the present disclosure may include: detecting that the content contains words, phrases, images, audio, etc. that do not match the age of the minor; detecting that the content contains text, images, audio, etc. that have discriminatory meanings against a certain group of people and have negative guidance for minors; detecting that the content contains text, images, audio, etc. that have semantics that have inappropriate guidance for minors.

[0070] The client 20 can be implemented as an electronic device such as a mobile phone, a tablet computer, a game console, a wearable device, a personal computer (PC), etc. Exemplarily, the client 20 can support the installation and operation of an application (Application, APP). In some embodiments, the application can be an independent application, that is, it can be directly run on the operating system of the client 20, without relying on other applications to run, such as short video APP, game APP, shopping APP, etc.; in other embodiments, the application can be a lightweight application that depends on other applications to run, which can be called a "applet", such as a game applet, a shopping applet, etc. running in an instant messaging APP. Exemplarily, the client 20 can be integrated with a display screen, which is configured to display the user interface (UI) of the client 20, and provide an interface for interaction and information exchange between the client 20 and the user, so that the client responds to the user's operation, switches the device operation mode, and collects and sends the content to be detected to the server, etc.

[0071] Understandably, Figure 1 The server 10 and the client 20 are both schematic diagrams at the logical function level. In actual implementation, the client 20 may include at least one electronic device entity, and the server 10 may include at least one server device entity. This is not limited here.

[0072] It should be noted that the monitoring model described in the embodiment of the present disclosure supports the detection of content in multiple modes. Taking the content of any mode as text, image or audio as an example, the structure of the monitoring model can be as follows: Figure 2 See Figure 2 The monitoring model may include a modality recognition module, a text detection module, an audio detection module, an image detection module and an intention prediction module, and the text detection module, the audio detection module and the image detection module are all detection modules.

[0073] Understandably, Figure 2 The monitoring model shown is an illustrative implementation of the present disclosure, and does not limit the implementation of the monitoring model in the embodiments of the present disclosure. In some other implementations, the content monitoring system may also include more or fewer algorithm modules. This is not limited here.

[0074] The modality recognition module can be used to identify the modality type of the input content, and when the input content includes content of multiple modalities, the input content is classified according to the modality, so as to transmit the content of each modality to the matching algorithm module for content detection. Exemplarily, the modality recognition module may include a convolutional neural network (CNN), multimodal parallel multitask learning (MPML), a multimodal deep Bellman machine (MDBM), etc.

[0075] The text detection module can be used to detect whether the input text meets the preset alarm conditions. Exemplarily, it can be used to detect whether the semantics of the input text meets the preset alarm conditions, and detect whether the words or phrases contained in the input text meet the preset alarm conditions. Exemplarily, the text detection module may include a bidirectional encoder representation (Bidirectional Encoder Representations from Transformers, BERT) model and a large language model (Large Language Models, LLMs), etc., wherein LLMs may include a pre-trained generative transform (Generative Pre-trained Transformer, GPT) series model or other large language models, etc.

[0076] The audio detection module can be used to detect whether the input audio meets the preset alarm conditions. Exemplarily, it can be used to detect whether the semantics of the input audio content meets the preset alarm conditions, and detect whether the emotion expressed by the input audio meets the preset alarm conditions. In some embodiments, the audio detection module may include an audio-to-text module, so that the input audio is converted into text, and then the text obtained after the conversion is input into the text detection module to detect the semantics of the input audio content. Exemplarily, the audio detection module may include automatic speech recognition (Automatic Speech Recognition, ASR), vocoder (Vocoder), hidden Markov model (HiddenMarkov Model, HMM), deep neural network (Deep Neural Networks, DNN) and recurrent neural network (Recurrent Neural Networks, RNN), etc.

[0077] The image detection module can be used to detect whether the input image contains content that meets the preset alarm conditions, such as violent image content. Exemplarily, the image detection module can include YOLO (You Only Look Once) algorithm, CNN and Faster R-CNN, etc.

[0078] In an actual implementation scenario, for the above-mentioned text detection module, audio detection module and image detection module, at least one detection module can be called according to the modality of the content to be detected specifically contained in the content set to be detected. For example, in an optional implementation scenario, the content set to be detected only contains text, then in this implementation scenario, only the text detection module can be called to detect whether the input text meets the alarm condition; for another example, in an optional implementation scenario, the content set to be detected contains text and audio, then in this implementation scenario, the text detection module can be called to detect whether the input text meets the alarm condition, and the audio detection module can be called to detect whether the input audio meets the alarm condition. Optionally, in the scenario of calling at least two detection modules, the at least two detection modules can be called to perform the detection operation in parallel.

[0079] The intention prediction module can be used to predict the user's operation intention based on the content changes caused by the user's continuous operations on the client, so as to further predict whether the user's intention involves potential threats to the network content. Exemplarily, the intention prediction module can include a long short-term memory network (Long Short-Term Memory, LSTM), a gated recurrent unit (Gated Recurrent Unit, GRU) and a Transformer, etc.

[0080] It should be noted that in some embodiments, the intention prediction module can be called to process the information output by the modality recognition module to predict the user's intention. In other embodiments, the intention prediction module can be called to further process the features output by the text detection module, the audio detection module, and the image detection module to predict the user's intention.

[0081] It should be understood that the algorithms and networks included in the modules of the above monitoring model are illustrative examples of the present disclosure and do not limit the modules in the embodiments of the present disclosure. The algorithms included in the modules in the embodiments of the present disclosure can be flexibly deployed according to the content detection purpose of the corresponding modality of the present disclosure and the connection relationship of each module. No limitation is given here.

[0082] Combine the following Figure 1 and Figure 2 An illustrative embodiment is used to describe a content monitoring method proposed according to an embodiment of the present disclosure.

[0083] like Figure 3As shown, the embodiment of the present disclosure provides a content monitoring method, which can be applied to a server, and the server can be Figure 1 The method comprises:

[0084] In step S11, a set of content to be detected is received from a client, where the set of content to be detected includes content to be detected in at least one modality.

[0085] Among them, the content set to be detected is collected after the client enters the monitoring mode in response to the user trigger. In some embodiments, the client can be implemented as an electronic device entity. Before receiving the content set to be detected from the client, the server can also receive a monitoring request sent by the client, and the monitoring request is sent by the electronic device entity after entering the monitoring mode in response to the user operation. Furthermore, the content set to be detected can be collected by the client at any time after entering the monitoring mode. The content to be detected in any mode is text, image or audio.

[0086] It should be noted that the content set to be detected may refer to the content output by all APPs running on the client. For example, the APPs running on the client include video APPs, and the content output by the video APP may include images, audio, and subtitles. Accordingly, the content set to be detected may include the video frame images, subtitle text, and audio corresponding to the video frames output by the video APP.

[0087] It should be understood that the modal types of the content to be detected contained in the content set to be detected can match the content being output by the client. In one optional implementation, the client is outputting the main text of a novel, for example, then the content set to be detected may only include content to be detected in the text mode; in another optional implementation, the client is outputting the interface of a game, for example, then the content set to be detected may include content to be detected in both image and audio modes.

[0088] In step S12, for any content to be detected in the set of content to be detected, a modality recognition module in the monitoring model is called to identify the modality of the content to be detected.

[0089] The structure of the monitoring model can be, for example, Figure 2 shown.

[0090] In combination with the above description of the content set to be detected, it can be known that the content set to be detected may contain content to be detected of at least one modality. In order to input the content to be detected of various modalities into the detection module used to detect the content to be detected of this modality, for any content to be detected in the content set to be detected, the modality recognition module in the monitoring model can be called to identify the modality of the content to be detected, so as to call the detection module in the monitoring model used to detect the content to be detected of this modality, to detect whether the content to be detected meets the preset alarm condition.

[0091] In step S13, after the target modality of the content to be detected is identified, a detection module in the monitoring model for detecting the content of the target modality is called to detect whether the content to be detected meets a preset alarm condition.

[0092] It should be understood that the features of the corresponding representation information of the content to be detected in different modalities are different, and the meanings represented by the representation information are also different. For example, the representation information of text can be obtained through the word features and sentence features of the text, and the representation information of text can represent the contextual semantics of the text; the representation information of an image can be obtained through the pixel features of the image, and the representation information of the image can represent the high-level semantic features of the image, that is, the semantics expressed by the content of the image; the representation information of audio can be obtained through the content, speech speed, intonation and volume of the audio, and the representation information of audio can represent the emotional semantics corresponding to the audio.

[0093] Based on this, the method specifically detects whether the content to be detected meets the preset alarm conditions in the following ways: if the content to be detected is text, the semantic features and semantic representation words of the content to be detected are extracted, and the semantic features and the semantic representation words are used as representation information; if the content to be detected is an image, the image semantic features of the content to be detected are extracted, and the image semantic features are used as representation information; if the content to be detected is audio, the text semantic features of the text corresponding to the content to be detected and the emotional features of the content to be detected are extracted, and the text semantic features and the emotional features are used as representation information; determine whether the content to be detected meets the alarm conditions based on the representation information.

[0094] The following combination Figure 2 The exemplary monitoring model shown in the figure illustrates the detection process of the to-be-detected content of different modalities. After the server calls the modality recognition module to identify the to-be-detected content set, the text in the to-be-detected content set can be input into the text detection module, the audio in the to-be-detected content set can be input into the audio detection module, and the image in the to-be-detected content set can be input into the image detection module.

[0095] Further, for text, the server can, for example, call the tokenizers of BERT or GPT in the text detection module to split the text into words or phrases. After that, on the one hand, it can determine whether the words or phrases obtained by tokenization include words in a preset violation word library. On the other hand, it can perform context semantic understanding on the text based on the words or phrases obtained by tokenization. If the words or phrases obtained by tokenization include words in the preset violation word library (for example, including the violation word "died"), or the result of semantic understanding is a representation information that meets the preset warning conditions (for example, the semantic representation of inducing consumption), then it is determined that the detection result of the text meets the preset warning conditions.

[0096] Performing context semantic understanding on the text based on the words or phrases obtained by tokenization can be executed by BERT or GPT. For example, if the BERT tokenizer is used to convert the text into WordPiece tokens and add markers at the beginning of the sentence and between sentences. Furthermore, the BERT model, through the Masked Language Model (MLM), randomly masks some words in the text and lets the BERT model predict these masked words, so as to learn the meaning representation of words based on the masked context and achieve bidirectional context understanding of the text. Another example is that if GPT is used, after tokenizing the text, it can call GPT to use an autoregressive method to predict the next word according to the previous context, so as to learn the coherence representation of the text. Furthermore, context understanding is performed based on the order of text characters from left to right.

[0097] In this way, not only can the explicit violation content in the text be identified, but also the potential violation orientation can be identified by understanding the context and semantic relationship, making the monitoring of text content more accurate.

[0098] For audio, on the one hand, the server can, for example, call ASR in the audio detection module to convert the audio into text, and then input the converted text into the text detection module for detection. The detection process of the text detection module is described in detail above and will not be elaborated here. On the other hand, the server can, for example, call Vocoder and DNN in the audio detection module to extract at least one of the intonation feature, volume feature, and speech rate feature of the audio, and detect whether the emotion expressed by the audio meets the preset warning conditions based on the extracted features of the audio. If the detection result of the text detection module meets the preset warning conditions, or the emotion expressed by the audio meets the preset content (for example, the emotion representation of inducing consumption, the emotion representation of being negative and厌世), then it is determined that the detection result of the audio meets the preset warning conditions.

[0099] It can be seen that by adopting this implementation method, not only can the illegal content in the text expressed by the audio be identified, but also the potential illegal orientation of the audio can be identified by analyzing the potential emotional orientation of the audio, making the monitoring of the audio more comprehensive.

[0100] For an image, the server can, for example, call the YOLO algorithm in the image detection module to detect and identify the objects contained in the image through a single forward propagation. After that, the Fast R-CNN region proposal network (RPN) can be called to generate candidate object regions from the image, each of which can contain an object. Furthermore, each candidate object region can be pooled by Fast R-CNN to extract a fixed-length feature vector, and each feature vector can be classified to determine the type of each object and the semantics of the image. Then, based on the type of each object and the semantics of the image, it can be determined whether the image contains an object that meets the alarm condition.

[0101] In step S14, if any mode of the to-be-detected content in the to-be-detected content set meets the alarm condition, an alarm message is sent to the client.

[0102] In the embodiment of the present disclosure, the server may integrate the results of whether each content to be detected in the content set to be detected meets the alarm condition, and use the result as the result of whether the content set to be detected meets the alarm condition.

[0103] In some embodiments, if the content to be detected in any mode in the content set to be detected meets the alarm condition (for example, the audio is negative and pessimistic representation information, and / or the image includes violent image content, etc.), the content set to be detected is considered to meet the alarm condition, and the server can send an alarm message to the client.

[0104] It can be seen that by adopting this implementation method, it is possible to detect whether the content of multiple modalities of the client meets the preset alarm conditions, and it is possible to monitor content with richer forms and improve the coverage of the monitored content. In addition, integrating the detection results of the content of multiple modalities as the monitoring results can improve the accuracy of monitoring and help improve the monitoring performance.

[0105] The above-mentioned content monitoring method embodiment is an implementation method for detecting whether the content set to be detected collected at one time violates the rules from the perspective of a content set to be detected. The disclosed embodiment can also support predicting user intentions based on multiple consecutive content sets to be detected, and the multiple consecutive content sets to be detected can be content generated by the client triggered by the user multiple times.

[0106] In combination with the above description, the server can start monitoring the client in response to the monitoring request of the client. In this monitoring phase of the client, based on the above embodiment, the server can also determine the total number of content sets to be detected that have been received. If the total number is greater than or equal to the preset number, predict the user intention based on the received content sets to be detected; determine whether the user intention has potential risks; if the user intention has potential risks, send risk reminder information to the client. It should be understood that the received content sets to be detected include the content sets to be detected.

[0107] It should be noted that this embodiment aims to predict the user's intention. Based on this, any one of the above-mentioned received content sets to be detected may be a set of content to be detected generated by the client in response to an operation of the user.

[0108] Combination Figure 2 In the illustrated monitoring model, if the total number of received content sets to be detected is greater than or equal to a preset number, the server may call the intention prediction module to predict the user intention based on the received content sets to be detected.

[0109] Exemplarily, for any modality, the intention prediction module can be called to obtain representation information of the content to be detected of the modality in each content set to be detected of the received content set to be detected; based on the receiving timing of each content set to be detected of the received content set to be detected, the latent features of the representation information of the content to be detected of the modality are extracted, and the latent features are used to characterize the changes in the content to be detected of the modality according to the receiving timing; and the user's intention towards the content to be detected of the modality is predicted based on the latent features.

[0110] Among them, latent features usually refer to features that cannot be directly presented, but can be inferred through the changing trend of features, or the meaning representation that can be extracted by mapping features to other dimensions. In the implementation scenario of the present disclosure, the intention of user behavior cannot be directly inferred through the set of content to be detected collected each time, but the user's preference for operation behavior can be obtained through the content changes generated after the user triggers multiple times in a row, and then the user's potential intention can be predicted.

[0111] For example, in one example, after starting content monitoring of a client entity, three sets of content to be detected of the client entity are obtained one by one. The first set of content to be detected includes an image of a game interface, the second set of content to be detected includes images and texts of game equipment, and the third set of content to be detected includes images and texts of a payment interface. Taking image content as an example, according to the order of the three sets of content to be detected, the image change process is the game interface, the image of game equipment, and the payment interface, wherein the semantics represented by the game interface is, for example, the process of playing games, the semantics represented by the image of game equipment is, for example, the process of selecting game equipment, and the semantics represented by the payment interface is, for example, payment. Further, the intention prediction module can predict that the user may have the intention to recharge the game or purchase game equipment based on the potential connection of the three images. Furthermore, the server can obtain the threat of financial loss in the user's intention and send risk reminder information that may cause financial loss to the client.

[0112] It can be seen that by adopting this implementation method, a deep learning model can also be used to predict potential threats based on the interaction between the user and the client, thereby further improving the user experience.

[0113] In other embodiments, if the content to be detected is text, after the text detection module segments the text, if it is determined that the semantic representation word is a word that meets the alarm condition, it is detected whether the preset violation word library contains the semantic representation word; if the preset violation word library does not contain the semantic representation word, the semantic representation word is added to the preset violation word library.

[0114] In addition, the server can also add the content set to be detected and the detection results of each content to be detected in the content set to be detected to a preset training sample library to obtain an updated training sample library; wherein the content set to be detected is used as a training sample, and the detection results are used as labels of the training samples. Furthermore, the server can adjust the parameters of the monitoring model based on the updated training sample library to obtain an updated monitoring model.

[0115] In this way, the technical solution of the embodiment of the present disclosure supports adaptive updating of the sample library and optimization of the monitoring model based on the updated sample library, thereby supporting continuous learning and updating of the monitoring model and being able to flexibly adapt to scenarios with diverse content presentation forms.

[0116] Correspondingly, such as Figure 4 As shown, the embodiment of the present disclosure provides another content monitoring method, which can be applied to a client, and the client can be Figure 1 The method comprises the following steps:

[0117] Step S21, when entering the monitoring mode, in response to the user's operation, collecting the to-be-detected content set of the client, wherein the to-be-detected content set includes to-be-detected content of at least one modality.

[0118] The content to be detected in any mode is text, image or audio, which will not be described in detail here.

[0119] Exemplarily, the client can be implemented as an electronic device entity, which can send a monitoring request to the server after entering the monitoring mode in response to a user operation, so that the server starts monitoring the electronic device entity. Further, after entering the monitoring mode, the client collects the network content being output by the client each time it receives a user operation.

[0120] Step S22: sending the content set to be detected to a server.

[0121] Among them, each time the client collects and outputs network content, the collected network content can be sent to the server as a set of content to be detected, so that the server can call the modality recognition module in the monitoring model to identify the modality of the content to be detected for any content to be detected in the set of content to be detected; and after identifying the target modality of the content to be detected, call the detection module in the monitoring model for detecting the content of the target modality to detect whether the content to be detected meets the preset alarm condition.

[0122] An example implementation of the monitoring model can be found in Figure 2 For the description of the server calling the monitoring model to detect the content to be detected, please refer to the description of the above embodiment and will not be repeated here.

[0123] Step S23, receiving alarm information from the server when the to-be-detected content of any mode in the to-be-detected content set meets the alarm condition.

[0124] In summary, the embodiment of the present disclosure pre-sets a monitoring model for detecting client content. The monitoring model may include a modality recognition module and a plurality of detection modules. The modality recognition module and the plurality of detection modules may be obtained based on deep learning, so that they are not restricted by fixed content and can flexibly adapt to the scenario of updated content presentation form through semantic understanding. Further, after the client enters the monitoring mode, in response to the user trigger, the content to be detected of at least one modality can be collected. Correspondingly, the server can call the monitoring model to identify the modality of the content to be detected of each modality, and detect whether the content to be detected meets the preset alarm conditions. When the content to be detected of any modality meets the alarm conditions, an alarm message is sent to the client. Among them, the content to be detected of any modality is text, image or audio. It can be seen that the embodiment of the present disclosure can monitor content with richer forms, which is conducive to improving monitoring performance.

[0125] correspond Figure 3 The present disclosure also provides a content monitoring device, which can be deployed in Figure 1 The server 10 is used to perform the operations of the content monitoring method performed by the server in any of the above embodiments, such as Figure 5A As shown, the device comprises:

[0126] The receiving unit 501 is used to receive a set of content to be detected from a client, wherein the set of content to be detected includes content to be detected in at least one mode, and the content to be detected in any mode is text, image or audio; the set of content to be detected is collected by the client in response to a user operation after entering a monitoring mode;

[0127] The identification unit 502 is used to call the modality identification module in the monitoring model to identify the modality of any content to be detected in the set of content to be detected;

[0128] The detection unit 503 is used to, after identifying the target mode of the content to be detected, call the detection module in the monitoring model for detecting the content of the target mode to detect whether the content to be detected meets the preset alarm condition;

[0129] The sending unit 504 is configured to send an alarm message to the client if any mode of the to-be-detected content in the to-be-detected content set meets the alarm condition.

[0130] In the embodiment of the present disclosure, for any content to be detected, the detection unit 503 is also used to extract semantic features and semantic representation words of the content to be detected if the content to be detected is text, and use the semantic features and the semantic representation words as representation information; if the content to be detected is an image, extract image semantic features of the content to be detected, and use the image semantic features as representation information; if the content to be detected is audio, extract text semantic features of the text corresponding to the content to be detected and emotional features of the content to be detected, and use the text semantic features and the emotional features as representation information; determine whether the content to be detected meets the alarm condition based on the representation information.

[0131] In the embodiment of the present disclosure, the device further includes: an adding unit. If the content to be detected is text, the detection module 502 is further used to detect whether the preset violation word library contains the semantic representation word if it is determined that the semantic representation word is a word that meets the alarm condition. The adding module is used to add the semantic representation word to the preset violation word library if the preset violation word library does not contain the semantic representation word.

[0132] In the embodiment of the present disclosure, the device further includes: a judgment unit, a prediction unit, and a determination unit. The judgment unit is used to judge the total number of received content sets to be detected; the prediction unit is used to predict the user intention based on the received content sets to be detected if the total number is greater than or equal to a preset number; the determination unit is used to determine whether the user intention has a potential risk; and the sending unit 504 is used to send risk reminder information to the client if the user intention has a potential risk.

[0133] In an embodiment of the present disclosure, the prediction unit is further used to obtain, for any modality, representation information of the content to be detected in each content set to be detected of the received content set to be detected; based on the reception timing of each content set to be detected of the received content set to be detected, extract latent features of the representation information of the content to be detected in the modality, the latent features being used to represent changes in the content to be detected in the modality according to the reception timing; and predict the user's intention towards the content to be detected in the modality based on the latent features.

[0134] In the disclosed embodiment, the device further includes: a training unit. The adding unit is further used to add the to-be-detected content set and the detection results of each to-be-detected content in the to-be-detected content set to a preset training sample library to obtain an updated training sample library; wherein the to-be-detected content set is used as a training sample, and the detection results are used as labels of the training samples; the training unit is used to adjust the parameters of the monitoring model based on the updated training sample library to obtain an updated monitoring model.

[0135] See also Figure 5B The present disclosure also provides a content monitoring device, which can be deployed in Figure 1 The client 20 is used to perform the operations of the content monitoring method performed by the client in any of the above embodiments, such as Figure 5B As shown, the device comprises:

[0136] The collection unit 511 is used to collect the to-be-detected content set of the client in response to the user's operation when entering the monitoring mode, wherein the to-be-detected content set includes to-be-detected content of at least one modality, and the to-be-detected content of any modality is text, image or audio;

[0137] The sending unit 512 is used to send the set of content to be detected to the server, so that the server calls the modality identification module in the monitoring model to identify the modality of the content to be detected for any content to be detected in the set of content to be detected; and after identifying the target modality of the content to be detected, calls the detection module in the monitoring model for detecting the content of the target modality to detect whether the content to be detected meets the preset alarm condition;

[0138] The receiving unit 513 is configured to receive an alarm message from the server when the content to be detected in any mode in the content to be detected set meets the alarm condition.

[0139] The content monitoring device provided in the above-mentioned embodiment of the present disclosure and the content monitoring method provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0140] The present disclosure also provides an electronic device to perform the above content monitoring method. Figure 6 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 6 As shown, the electronic device 6 includes: a processor 600, a memory 601, a bus 602 and a communication interface 603, wherein the processor 600, the communication interface 603 and the memory 601 are connected via the bus 602; the memory 601 stores a computer program that can be run on the processor 600, and when the processor 600 runs the computer program, the content monitoring method provided in any of the aforementioned embodiments of the present disclosure is executed.

[0141] It should be pointed out that Figure 6 The electronic device shown can be implemented as Figure 1In the server device or client device, when the electronic device is implemented as a server device, the aforementioned monitoring model can also be stored in the memory 601. When the electronic device is implemented as a client device, the electronic device can also include a display.

[0142] The memory 601 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the virtual devices of the system is realized through at least one communication interface 603 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0143] The bus 602 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store programs, and the processor 600 executes the programs after receiving execution instructions. The content monitoring method disclosed in any implementation of the aforementioned embodiment of the present disclosure may be applied to the processor 600, or implemented by the processor 600.

[0144] The processor 600 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 600 or the instruction in the form of software. The above processor 600 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 601, and the processor 600 reads the content in the memory 601 and completes the steps of the above method in combination with its hardware.

[0145] The electronic device provided by the embodiment of the present disclosure and the content monitoring method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0146] The present disclosure also provides a computer-readable storage medium corresponding to the content monitoring method provided in the above embodiment. Figure 7 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the content monitoring method provided by any of the aforementioned embodiments will be executed.

[0147] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0148] The computer-readable storage medium provided by the above-mentioned embodiment of the present disclosure and the content monitoring method provided by the embodiment of the present disclosure are based on the same inventive concept, and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0149] Although the optional embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0150] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present disclosure. It should be understood that the above description is only the specific implementation method of the present disclosure and is not intended to limit the protection scope of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present disclosure should be included in the protection scope of the present invention.

Claims

1. A content monitoring method, characterized in that: Applied to the server, the method includes: Receive a set of content to be detected from a client, wherein the set of content to be detected includes content to be detected in at least one mode, and the content to be detected in any mode is text, image or audio; the set of content to be detected is collected in response to a user operation after the client enters a monitoring mode; For any content to be detected in the set of content to be detected, calling a modality identification module in the monitoring model to identify the modality of the content to be detected; After identifying the target mode of the content to be detected, calling the detection module in the monitoring model for detecting the content of the target mode to detect whether the content to be detected meets the preset alarm condition; If any mode of the to-be-detected content in the to-be-detected content set meets the alarm condition, an alarm message is sent to the client.

2. The method according to claim 1, characterized in that The method specifically detects whether the content to be detected meets the preset alarm condition in the following manner: If the content to be detected is text, extracting semantic features and semantic representation words of the content to be detected, and using the semantic features and the semantic representation words as representation information; If the content to be detected is an image, extracting image semantic features of the content to be detected, and using the image semantic features as representation information; If the content to be detected is audio, extracting text semantic features of the text corresponding to the content to be detected and emotional features of the content to be detected, and using the text semantic features and the emotional features as representation information; Determine whether the content to be detected meets the alarm condition according to the characterization information.

3. The method according to claim 2, characterized in that If the content to be detected is text, the method further includes: If it is determined that the semantic representation word is a word that meets the alarm condition, detecting whether the preset violation word library contains the semantic representation word; If the preset violation word library does not include the semantic representation word, the semantic representation word is added to the preset violation word library.

4. The method according to any one of claims 1 to 3, characterized in that Also includes: Determine the total number of received content sets to be detected; If the total number is greater than or equal to the preset number, predicting the user intention according to the received set of content to be detected; Determining whether the user's intention is potentially risky; If the user's intention involves potential risks, a risk reminder message is sent to the client.

5. The method according to claim 4, characterized in that The predicting the user intention according to the received content set to be detected includes: For any modality, obtaining representation information of the content to be detected of the modality in each content set to be detected of the received content set to be detected; Extracting latent features of the characterization information of the content to be detected in the modality based on the receiving timing of each of the received content sets to be detected, wherein the latent features are used to characterize changes of the content to be detected in the modality according to the receiving timing; Predict the user's intention towards the content to be detected in the modality according to the latent features.

6. The method according to claim 1, characterized in that Also includes: Adding the to-be-detected content set and the detection results of each to-be-detected content in the to-be-detected content set to a preset training sample library to obtain an updated training sample library; wherein the to-be-detected content set is used as a training sample, and the detection results are used as labels of the training sample; The parameters of the monitoring model are adjusted based on the updated training sample library to obtain an updated monitoring model.

7. A content monitoring method, characterized in that: Applied to a client, the method comprises: In the case of entering the monitoring mode, in response to the user's operation, collecting the to-be-detected content set of the client, wherein the to-be-detected content set includes to-be-detected content of at least one modality, and the to-be-detected content of any modality is text, image or audio; The set of content to be detected is sent to the server, so that the server calls the modality identification module in the monitoring model to identify the modality of any content to be detected in the set of content to be detected; and after identifying the target modality of the content to be detected, calls the detection module in the monitoring model for detecting the content of the target modality to detect whether the content to be detected meets the preset alarm condition; When the content to be detected in any mode in the set of content to be detected meets the alarm condition, receiving alarm information from the server.

8. A content monitoring device, characterized in that: Applied to the server, the device comprises: A receiving unit, configured to receive a set of content to be detected from a client, wherein the set of content to be detected includes content to be detected in at least one mode, and the content to be detected in any mode is text, image or audio; the set of content to be detected is collected in response to a user's operation after the client enters a monitoring mode; an identification unit, configured to call a modality identification module in the monitoring model to identify a modality of any content to be detected in the set of content to be detected; A detection unit, configured to, after identifying the target mode of the content to be detected, call a detection module in the monitoring model for detecting the content of the target mode to detect whether the content to be detected meets a preset alarm condition; The sending unit is used to send an alarm message to the client if any mode of the to-be-detected content in the to-be-detected content set meets the alarm condition.

9. A content monitoring device, characterized in that: Applied to a client, the device comprises: a collection unit, configured to collect a set of to-be-detected content of the client in response to a user operation when entering the monitoring mode, wherein the set of to-be-detected content includes to-be-detected content of at least one modality, and the to-be-detected content of any modality is text, image or audio; A sending unit, configured to send the set of content to be detected to a server, so that the server calls the modality identification module in the monitoring model to identify the modality of any content to be detected in the set of content to be detected; and after identifying the target modality of the content to be detected, calls the detection module in the monitoring model for detecting the content of the target modality to detect whether the content to be detected meets a preset alarm condition; The receiving unit is used to receive the alarm information from the server when the content to be detected in any mode in the content to be detected set meets the alarm condition.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the computer program to implement the method according to any one of claims 1 to 7.

Images