A general content detection method, device and electronic equipment

Abstract

Embodiments of the present application disclose a general content detection method, device and electronic equipment; semantic analysis can be performed on a target statement to be detected to obtain a target semantic vector corresponding to the target statement, wherein the target statement is from user-generated content, similarity between the target semantic vector and a reference semantic vector is determined, wherein the reference semantic vector is a semantic vector corresponding to a reference statement, the reference statement is from reference user-generated content, a reference semantic vector with similarity greater than a preset similarity threshold is determined as a target reference semantic vector, the number of reference semantic vectors and the number of target reference semantic vectors are obtained, and if the ratio between the number of target reference semantic vectors and the number of reference semantic vectors is greater than a preset value, it is determined that the target statement belongs to a general statement; thus, content with lower quality for content reading users can be screened from user-generated content, and the auditing efficiency of user-generated content can be improved on the basis of ensuring auditing accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, and in particular to a general content detection method and device and electronic equipment. BACKGROUND

[0002] With the rapid development of the Internet, more and more users produce their own content through various platforms. Due to the large number of users, how to efficiently and accurately review the content produced by the users is very important.

[0003] At present, there is a kind of general content in the content produced by the users. The general content is different from plagiarism and text not matching the topic. This kind of content can be considered to conform to the theme, and there is no plagiarism of other content, and the sentence is also relatively smooth, but it can not provide much help to the reader. The user reading the general content may already have the relevant knowledge in the general content, so this kind of content is of low quality to the reading user and needs to be detected. SUMMARY

[0004] The embodiments of the present application provide a general content detection method, device and electronic equipment, which can screen out the content of low quality to the reading user from the massive user-produced content, improve the review efficiency of the user-produced content on the basis of ensuring the review accuracy.

[0005] The embodiments of the present application provide a general content detection method, comprising:

[0006] performing semantic analysis on a target sentence to be detected to obtain a target semantic vector corresponding to the target sentence, wherein the target sentence is from user-produced content;

[0007] determining the similarity between the target semantic vector and a reference semantic vector, wherein the reference semantic vector is a semantic vector corresponding to a reference sentence, and the reference sentence is from reference user-produced content;

[0008] determining the reference semantic vector with a similarity greater than a preset similarity threshold as a target reference semantic vector, and obtaining the number of the reference semantic vectors and the number of the target reference semantic vectors;

[0009] if the ratio between the number of the target reference semantic vectors and the number of the reference semantic vectors is greater than a preset value, determining that the target sentence belongs to a general sentence.

[0010] Correspondingly, the embodiments of the present application also provide a general content detection device, comprising:

[0011] a semantic analysis unit configured to perform semantic analysis on a target sentence to be detected to obtain a target semantic vector corresponding to the target sentence, wherein the target sentence is from user-produced content;

[0012] a similarity calculation unit, configured to determine a similarity between the target semantic vector and a reference semantic vector, the reference semantic vector being a semantic vector corresponding to a reference sentence from a reference user's production content;

[0013] a target vector determination unit, configured to determine a reference semantic vector with a similarity greater than a preset similarity threshold as a target reference semantic vector, and to obtain a quantity of the reference semantic vectors and a quantity of the target reference semantic vectors;

[0014] a general sentence determination unit, configured to determine that the target sentence belongs to a general sentence if a ratio between the quantity of the target reference semantic vectors and the quantity of the reference semantic vectors is greater than a preset value.

[0015] In an optional example, the similarity calculation unit is configured to perform semantic relationship analysis on the target semantic vector and each of the reference semantic vectors to obtain a semantic relationship vector corresponding to the target semantic vector and each of the reference semantic vectors.

[0016] The similarity between the target semantic vector and each of the reference semantic vectors is determined by performing semantic similarity analysis on each of the semantic relationship vectors.

[0017] In an optional example, the reference semantic vector is from a node semantic vector, and the similarity calculation unit further includes a vector searching unit configured to obtain a pre-established vector index file, the vector index file including nodes corresponding to each of the node semantic vectors and a node distribution relationship indicating a distribution relationship of each of the node semantic vectors.

[0018] The target semantic vector is represented as a target node in the vector index file based on a node creation manner in the vector index file.

[0019] A reference node matching the target node is determined according to the target node and the vector index file.

[0020] A reference semantic vector corresponding to the reference node is determined from the node semantic vector based on a corresponding relationship between the node semantic vector and the node.

[0021] In an optional example, the vector searching unit further includes an index construction unit configured to perform semantic analysis on a first reference sentence to obtain a node semantic vector corresponding to the first reference sentence.

[0022] The nodes corresponding to each of the node semantic vectors are created according to the distribution relationship of each of the node semantic vectors, and a distribution relationship between the nodes is associated with the distribution relationship of the node semantic vectors.

[0023] construct the vector index file based on the nodes corresponding to the node semantic vectors, the correspondence between the node semantic vectors and the nodes, and the distribution relationship between the nodes.

[0024] In an optional example, the index construction unit is followed by a first updating unit configured to execute the step of performing semantic analysis on the target sentence as a first reference sentence to obtain a node semantic vector corresponding to the first reference sentence, and obtain a new vector index file.

[0025] Detect the number of target sentences to be detected. If the number of target sentences to be detected is less than a preset value, replace the vector index file currently in use with the new vector index file.

[0026] In an optional example, the index construction unit is followed by a second updating unit configured to perform semantic analysis on the target sentence as a new first reference sentence to obtain a new node semantic vector corresponding to the new first reference sentence.

[0027] Based on a preset vector node calculation method, calculate a new node corresponding to the new node semantic vector.

[0028] According to the new node and the distribution relationship between the nodes in the vector index, obtain a new node distribution relationship between the new node and the existing nodes.

[0029] Based on the new node semantic vector and the new node distribution relationship, update the vector index file.

[0030] In an optional example, the semantic analysis unit is configured to perform semantic relationship analysis on the target semantic vector and each reference semantic vector by using a semantic relationship representation model.

[0031] The semantic analysis unit is followed by a model training unit configured to obtain sample target semantic vectors and sample reference semantic vectors, and actual similarity of the sample target semantic vectors and the sample reference semantic vectors.

[0032] Based on a preset vector connection mark, perform vector splicing on the sample target semantic vectors and the sample reference semantic vectors to obtain spliced vectors.

[0033] Perform semantic relationship analysis on the spliced vectors by using a to-be-trained semantic relationship representation model to obtain sample semantic relationship vectors corresponding to the sample target semantic vectors and the sample reference semantic vectors.

[0034] According to the sample semantic relation vector, semantic approximation degree analysis is performed to obtain a predicted similarity between the sample target semantic vector and the sample reference semantic vector;

[0035] According to the predicted similarity and the actual similarity, a loss of the semantic relation representation model to be trained is determined.

[0036] According to the loss, a model parameter of the semantic relation representation model to be trained is adjusted to obtain a trained semantic relation representation model.

[0037] In an optional example, the user production content includes at least one sentence, and the general content detection apparatus further includes a general content determination unit configured to determine another sentence from the user interaction content as a target sentence to be detected, and return to perform the semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence until all the sentences in the user production content participate in the general content detection.

[0038] A ratio between a number of target reference semantic vectors corresponding to each of the sentences in the user interaction content and a number of reference semantic vectors is recorded.

[0039] According to the ratio corresponding to each of the sentences, a general probability corresponding to the user production content is calculated, and if the general probability is greater than a preset probability, the user production content is determined as general content.

[0040] Correspondingly, the embodiment of the present application further provides an electronic device, including a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the operations in any one of the general content detection methods provided by the embodiment of the present application.

[0041] In addition, the embodiment of the present application further provides a storage medium, and the storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in any one of the general content detection methods provided by the embodiment of the present application.

[0042] By adopting the scheme of the embodiment of the present application, semantic analysis can be performed on a target sentence to be detected, a target semantic vector corresponding to the target sentence is obtained, the target sentence is from user-generated content, similarity between the target semantic vector and a reference semantic vector is determined, the reference semantic vector is a semantic vector corresponding to a reference sentence, the reference sentence is from reference user-generated content, a reference semantic vector with similarity greater than a preset similarity threshold is determined as a target reference semantic vector, the number of the reference semantic vectors and the number of the target reference semantic vectors are obtained, and if the ratio between the number of the target reference semantic vectors and the number of the reference semantic vectors is greater than a preset value, it is determined that the target sentence is a general sentence. Thus, in the embodiment, semantic analysis is performed on a sentence as a granularity, the target semantic vector corresponding to the target sentence is compared with the plurality of reference semantic vectors in terms of semantic similarity, if the target sentence has semantic similarity with the plurality of reference sentences, it can be determined that the target sentence can be used in different content production processes, user-generated content without plagiarism but with small amount of information and unable to provide help for content reading users is detected, and the content review efficiency of the user-generated content is improved on the basis of ensuring the accuracy of the content review. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0044] Figure 1 is a scene diagram of a general content detection method provided by the embodiment of the present application;

[0045] Figure 2 is a flowchart of a general content detection method provided by the embodiment of the present application;

[0046] Figure 3 is a process diagram of obtaining a semantic relationship vector between a target semantic vector and a reference semantic vector by a semantic relationship representation model provided by the embodiment of the present application;

[0047] Figure 4 is another flowchart of a general content detection method provided by the embodiment of the present application;

[0048] Figure 5 is a node distribution relationship diagram of a graph structure provided by the embodiment of the present application;

[0049] Figure 6 is a node distribution relationship diagram of a tree structure provided by the embodiment of the present application;

[0050] Figure 7 is another node distribution relationship diagram of a tree structure provided by an embodiment of the present application;

[0051] Figure 8 is a result example diagram of general content detection provided by an embodiment of the present application;

[0052] Figure 9 is another result example diagram of general content detection provided by an embodiment of the present application;

[0053] Figure 10 is a page diagram when a user production content execution non-display strategy is executed for general content provided by an embodiment of the present application;

[0054] Figure 11 is another flow diagram of a general content detection method provided by an embodiment of the present application;

[0055] Figure 12 is a structure diagram of a general content detection device provided by an embodiment of the present application;

[0056] Figure 13 is another structure diagram of a general content detection device provided by an embodiment of the present application;

[0057] Figure 14 is another structure diagram of a general content detection device provided by an embodiment of the present application;

[0058] Figure 15 is a structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the protection scope of the present application.

[0060] The embodiments of the present application provide a general content detection method, device, electronic device and storage medium. Specifically, the embodiments of the present application provide a general content detection method suitable for a general content detection device, which can be integrated in an electronic device.

[0061] The electronic device can be a terminal and the like, including but not limited to a mobile terminal and a fixed terminal. For example, the mobile terminal includes but is not limited to a smart phone, a smart watch, a tablet computer, a notebook computer, a smart vehicle and the like. The fixed terminal includes but is not limited to a desktop computer, a smart television and the like.

[0062] The electronic device can also be a server, which can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms, but is not limited thereto.

[0063] The general content detection method of the embodiment of the application can be implemented by a server or by a terminal and a server together.

[0064] The general content detection method will be described below by taking the case of being implemented by a terminal and a server together.

[0065] As shown in Figure 1 The general content detection system provided by the embodiment of the application includes a terminal 10 and a server 20, etc. The terminal 10 and the server 20 are connected through a network, such as a wired or wireless network connection, etc. The terminal 10 can exist as a terminal for a user to send a target sentence to be detected to the server 20.

[0066] The terminal 10 can be a terminal for a user to upload user-generated content, which is used to send the user-generated content to the server 20. The server 20 can process the user-generated content to obtain a target sentence to be detected after obtaining the user-generated content.

[0067] The server 20 can be used to perform semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence. The target sentence is from the user-generated content. The similarity between the target semantic vector and a reference semantic vector is determined, wherein the reference semantic vector is a semantic vector corresponding to a reference sentence, and the reference sentence is from a reference user-generated content. The reference semantic vector with a similarity greater than a preset similarity threshold is determined as a target reference semantic vector. The number of reference semantic vectors and the number of target reference semantic vectors are obtained. If the ratio between the number of target reference semantic vectors and the number of reference semantic vectors is greater than a preset value, it is determined that the target sentence belongs to a general sentence.

[0068] In one example, after the server 20 performs general sentence detection on a sentence in the user-generated content, another sentence in the user-generated content can be determined as a target sentence to be detected for general sentence detection. The proportion of general sentences between the number of target sentences belonging to general sentences and the number of all sentences in the user-generated content is calculated. If the proportion of general sentences is greater than a preset proportion, it is determined that the user-generated content is general content.

[0069] The following will be described in detail respectively. It should be noted that the description order of the following embodiments is not as a limitation on the preferred order of the embodiments.

[0070] Embodiments of the present application will be described from the perspective of a general content detection device, which can be integrated in a server or terminal.

[0071] As shown in Figure 2 The specific flow of the general content detection method of the present embodiment can be as follows:

[0072] 201. Perform semantic analysis on the target statement to be detected to obtain a target semantic vector corresponding to the target statement, wherein the target statement is from user generated content.

[0073] In the present embodiment, user generated content (UGC) can refer to user original content, that is, the user displays or provides the content created by himself to other users through an Internet platform. The production subject of user generated content is generally an ordinary user, mainly for the purpose of sharing personal experiences and interests, and producing and disseminating content. For example, the content and comments published by the user on the social platform, personal video creation, etc. all belong to user generated content.

[0074] Among them, the general content can be considered as "universal content" different from plagiarism and off-topic, which can be considered as meeting the theme, not plagiarizing other content, and the sentence is also relatively smooth, but it can not provide much help to the reader. The user reading the general content may already have the relevant knowledge in the general content, so such content is of low quality for the reading user and needs to be detected.

[0075] Among them, the process of performing semantic analysis on the target statement to obtain the corresponding target semantic vector involves nature language processing (NLP) technology. Nature language processing is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science and mathematics. Therefore, the research in this field will involve natural language, that is, the language used in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph, etc.

[0076] The target sentence is one or more sentences in the user-generated content that need to be detected by the general content detection. The user-generated content can be user-generated content received by the server in real time, or content stored in the server for a period of time. That is, after the server receives the user-generated content, the server can store the user-generated content in the database. When a preset general content detection time (for example, 02:00 in the morning) is reached, the server can obtain user-generated content that meets the preset detection condition from the database to participate in the general content detection.

[0077] It can be understood that the general content as a whole can not be similar to the user-generated content of other users, and the user-generated content needs to be detected at the sentence granularity. Therefore, before the general content detection, the user-generated content needs to be preprocessed. That is, in an optional example, step 201 can include:

[0078] The user-generated content is segmented into sentences to obtain the sentences in the user-generated content.

[0079] When the user-generated content is segmented into sentences, the segmentation can be performed according to punctuation marks such as “.”, “?”, “!”, “,” and the like in the user-generated content.

[0080] In an example of some embodiments, the target semantic vector corresponding to the target sentence can be obtained by obtaining the word vector corresponding to the target sentence and performing calculation on the word vector.

[0081] The target sentence is segmented into words to obtain the words in the target sentence.

[0082] The words are mapped to word vectors corresponding to the words based on a preset vector mapping relationship.

[0083] The target semantic vector corresponding to the target sentence is obtained based on the calculation of the word vectors.

[0084] When the words are mapped to the word vectors, the mapping can also be realized by a semantic representation model, such as a Word2vec model, a continuous bag of words (CBOW) model, and the like.

[0085] Optionally, the semantic representation model can directly process the target sentence. For example, the semantic representation model can include a word segmentation processing module, a word vector representation module, and a semantic vector representation module.

[0086] Correspondingly, the step of “performing semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence” can include:

[0087] In the word segmentation processing module, the target sentence is processed by word segmentation to obtain the word segmentation of the target sentence.

[0088] Through the word vector representation module, each word segmentation is mapped to the corresponding word vector of each word segmentation.

[0089] Through the semantic vector representation module, each word vector is calculated to obtain the target semantic vector corresponding to the target sentence.

[0090] Among them, the step "calculating based on each word vector to obtain the target semantic vector corresponding to the target sentence" can include:

[0091] The average of each word vector is calculated to obtain the target semantic vector corresponding to the target sentence.

[0092] In some optional examples, the maximum value, minimum value, splicing, weighted calculation, etc. can also be taken for each word vector to obtain the target semantic vector.

[0093] 202, determine the similarity between the target semantic vector and the reference semantic vector, wherein the reference semantic vector is the semantic vector corresponding to the reference sentence, and the reference sentence comes from the reference user production content.

[0094] Among them, the generation method of the reference semantic vector is similar to that of the target semantic vector, and the embodiments of the present application do not repeat it.

[0095] Among them, the reference user production content can be obtained from the historical user production content of the content production and distribution platform. For example, the content production and distribution platform can be a question and answer platform, and the reference user production content can be the user reply under different questions in the question and answer platform, etc.

[0096] In order to ensure the universality of the target sentence, generally, the reference user production content can cover as much historical user production content of different topics as possible. For example, when the content production and distribution platform is a medical consultation website, the reference user production content can come from the user reply content of different questions in the medical fields of respiratory medicine, digestive medicine, stomatology, ophthalmology and / or obstetrics and gynecology, etc.

[0097] In some embodiments, the content production and distribution platform can be understood as any form of platform that can publish content, and the platform can also distribute the content published on its platform to other platforms. The content production and distribution platform includes a web page form platform, a client form platform, or a platform integrated in the client. For example, various news client, web page form search engine, etc.

[0098] It can be understood that the user interaction content published by the user may have different speaking preferences on different content production and distribution platforms. The reference user production content of the embodiment can be obtained from at least one content production and distribution platform, for example, can include content obtained from historical user production content of the current content production and distribution platform (the platform where the user production content is located), and can also include user production content obtained from other content production and distribution platforms, thereby improving the data quantity and diversity of the reference user production content, and further improving the detection effect on the general content.

[0099] In some embodiments, the similarity between the target semantic vector and the reference semantic vector can be calculated by calculating the Euclidean distance, Manhattan distance, included angle cosine, Pearson correlation coefficient, etc. between the target semantic vector and each reference semantic vector, and taking the calculated value as the similarity between the target semantic vector and the reference semantic vector.

[0100] In other embodiments, in order to better describe the similarity between the target sentence and the reference sentence from the semantic point of view, specifically, the step of "determining the similarity between the target semantic vector and the reference semantic vector" can include:

[0101] performing semantic relationship analysis on the target semantic vector and each reference semantic vector to obtain a semantic relationship vector corresponding to the target semantic vector and each reference semantic vector;

[0102] performing semantic similarity analysis on each semantic relationship vector to obtain the similarity between the target semantic vector and each reference semantic vector.

[0103] Wherein, when performing semantic relationship analysis on the target semantic vector and each reference semantic vector, the analysis can be performed according to the information such as the words included in the semantic vector, the syntax of the sentence, and the semantics of the sentence.

[0104] In some embodiments, in order to improve the accuracy of semantic relationship analysis, as shown in Figure 3 the analysis can be performed through a semantic relationship representation model, that is, the step of "performing semantic relationship analysis on the target semantic vector and each reference semantic vector" can include:

[0105] performing semantic relationship analysis on the target semantic vector and each reference semantic vector through a semantic relationship representation model.

[0106] Wherein, the model parameters of the semantic relationship representation model can be obtained by deep learning. The concept of deep learning is derived from the research of artificial neural networks. Deep learning forms more abstract high-level representation attribute categories or features by combining low-level features to discover distributed feature representation of data.

[0107] In an optional example, after obtaining the semantic relation vector, the semantic relation vector can be classified by a classifier or a classification model to obtain a relevance classification result of the target semantic vector and each reference semantic vector.

[0108] In an example, to improve the accuracy of the semantic relation representation model, the semantic relation representation model can be trained, and before the step of "performing semantic relation analysis on the target semantic vector and each reference semantic vector by the semantic relation representation model", the following steps can be further included:

[0109] Obtaining a sample target semantic vector and a sample reference semantic vector, and an actual similarity between the sample target semantic vector and the sample reference semantic vector;

[0110] Performing vector splicing on the sample target semantic vector and the sample reference semantic vector based on a preset vector splicing mark to obtain a spliced vector;

[0111] Performing semantic relation analysis on the spliced vector by the semantic relation representation model to be trained to obtain a sample semantic relation vector corresponding to the sample target semantic vector and the sample reference semantic vector;

[0112] Performing semantic similarity analysis according to the sample semantic relation vector to obtain a predicted similarity between the sample target semantic vector and the sample reference semantic vector;

[0113] Determining a loss of the semantic relation representation model to be trained according to the predicted similarity and the actual similarity;

[0114] Adjusting model parameters of the semantic relation representation model to be trained according to the loss to obtain a trained semantic relation representation model.

[0115] For example, the semantic relation representation model can be a Bidirectional Encoder Representation from Transformers (BERT), Word2vec, etc.

[0116] The loss of the semantic relation representation model under the current model parameters can be calculated according to the predicted similarity and the actual similarity by a loss function such as a cross-entropy loss function or a mean square error loss function, the model parameters of the semantic relation representation model are adjusted based on the loss, and then the step of "performing semantic relation analysis on the spliced vector by the semantic relation representation model to be trained to obtain a sample semantic relation vector corresponding to the sample target semantic vector and the sample reference semantic vector" is executed, the above steps are executed multiple times until a model training end condition is met, and a trained semantic relation representation model is obtained.

[0117] The model training end condition includes, but is not limited to, that the loss of the semantic relation representation model is not more than a preset loss threshold, or the number of times of training of the semantic relation representation model is not less than a preset minimum number threshold, or the loss of the semantic relation representation model tends to be stable (for example, the loss is no longer reduced), and the like.

[0118] It can be understood that, in some embodiments, the reference semantic vectors can be all semantic vectors that can be used for similarity comparison with the target semantic vector and are prepared by the technician in advance, and when the similarity of the target semantic vector and the reference semantic vector is determined, the similarity of the target semantic vector and all the reference semantic vectors can be calculated respectively.

[0119] In other embodiments, the technician can prepare semantic vectors that can be used for similarity comparison with the target semantic vector, but the number of the semantic vectors can be large, and in order to improve the calculation efficiency and save the calculation resources, the semantic vectors can be provided with vector indexes, and when the similarity of the target semantic vector and the reference semantic vector is determined, some semantic vectors closest to the target semantic vector can be found as the reference semantic vectors through the vector indexes. That is, the reference semantic vectors come from the node semantic vectors, and before step 202, the method can further include:

[0120] obtaining a pre-established vector index file, the vector index file including nodes corresponding to the node semantic vectors and node distribution relations indicating the distribution relations of the node semantic vectors;

[0121] based on the node creation mode in the vector index file, representing the target semantic vector as a target node in the vector index file;

[0122] determining a reference node matched with the target node according to the target node and the vector index file;

[0123] based on the correspondence between the node semantic vectors and the nodes, determining a reference semantic vector corresponding to the reference node from the node semantic vectors.

[0124] The node is a node in the vector index, and in different vector index file construction modes, the specific form of the node can be different. For example, the node can be directly represented by the node semantic vector, or the node can be a hyperplane when the vector space is divided into vectors, or the node can be a certain point in a multi-dimensional space, or the node can be a group of identification information for indicating the corresponding node semantic vector, and the like. The embodiments of the present application do not limit this.

[0125] In some examples, one node can correspond to only one node semantic vector, for example, as shown in FIG. 2, the node distribution relation in the vector index file is in the form of an index of a graph structure, and the node can be directly represented by the node semantic vector. Figure 5 ​

[0126] In some examples, one node can correspond to at least one node semantic vector, for example, as shown in FIG. 1, the distribution relationship of nodes in the vector index file can be considered in the form of a tree structure, and the nodes can be hyperplanes when the vector space is divided into vectors; or, as shown in FIG. 2, the distribution relationship of nodes in the vector index file can be considered in the form of a tree structure, and the nodes can be points in a multi-dimensional space, and the like. Figure 6 Figure 7 In some examples, one node can correspond to at least one node semantic vector, for example, as shown in FIG. 1, the distribution relationship of nodes in the vector index file can be considered in the form of a tree structure, and the nodes can be hyperplanes when the vector space is divided into vectors; or, as shown in FIG. 2, the distribution relationship of nodes in the vector index file can be considered in the form of a tree structure, and the nodes can be points in a multi-dimensional space, and the like.

[0127] Specifically, the vector index file can be created in the following way, before the step of "determining the reference node matching the target node according to the target node and the vector index", further comprising:

[0128] performing semantic analysis on the first reference sentence to obtain a node semantic vector corresponding to the first reference sentence;

[0129] creating nodes corresponding to the node semantic vectors according to the distribution relationship of the node semantic vectors, and associating the distribution relationship between the nodes with the distribution relationship of the node semantic vectors;

[0130] constructing the vector index file based on the nodes corresponding to the node semantic vectors, the correspondence between the node semantic vectors and the nodes, and the distribution relationship between the nodes.

[0131] It can be understood that when the similarity between the target semantic vector and all reference semantic vectors is calculated respectively, the reference semantic vector corresponding to the reference sentence can be the same as the first reference sentence used to construct the vector index file, or when the closest semantic vectors to the target semantic vector are found as reference semantic vectors through the vector index, the reference semantic vector corresponding to the reference sentence can be a part of the first reference sentence used to construct the vector index file.

[0132] In some examples, the distribution relationship between the nodes can indicate the positional relationship between the node semantic vectors, that is, before the step of "determining the reference node matching the target node according to the target node and the vector index", can include:

[0133] performing semantic analysis on the first reference sentence to obtain a node semantic vector corresponding to the first reference sentence;

[0134] performing vector space division on the vector space where the node semantic vectors are located according to the spatial positions of the node semantic vectors to determine the vector subspaces where the node semantic vectors are located;

[0135] establishing the correspondence between the node semantic vectors and the vector subspaces;

[0136] ​Based on the node semantic vectors and the corresponding relationship between each node semantic vector and each vector subspace, a vector index file is established.

[0137] For example, in the form of graph index, the edges can be spatially layered according to the characteristic radius of each node semantic vector by adopting a layered structure, so that the average degree of each node in all layers becomes a constant. For another example, in the form of tree index, the variance of each dimension can be calculated, and then the dimension with the largest variance is selected to start the division, so that the depth of the tree is as small as possible. For another example, in the form of clustering index, the space can be continuously divided by the plane of the selected two centroids, and finally the sample data in each divided subspace is limited within K. The value of K can be set by the technician according to the actual situation, and the present application does not limit this.

[0138] Correspondingly, in order to improve the similarity calculation efficiency and save computing resources, the step of "determining the similarity between the target semantic vector and the reference semantic vector" can include:

[0139] determining the vector subspace corresponding to the target semantic vector, the vector subspace including at least one node semantic vector;

[0140] calculating the vector distance between the semantic vector and each reference comparison vector in the vector subspace;

[0141] determining the reference semantic vector from the node semantic vectors based on the vector distance.

[0142] For example, when searching, a certain (several) subspace is quickly locked by a certain method, and then traversal is performed in the (several) subspace, so that we can obtain the K vectors closest to the query vector in real time.

[0143] It can be understood that the user-generated content such as replies or articles has a certain timeliness, and the detection effect is easy to degrade. In order to ensure the effect of general content detection, the node semantic vector should be updated in time. In some vector index construction processes, the constructed vector index file does not support immediate update, for example, the vector index file can be constructed according to the KD-tree algorithm, Annoy algorithm, etc. At this time, the new node semantic vector needs to be merged with the original node semantic vector for vector index construction. That is, after the step of "performing semantic analysis on the target sentence to obtain the target semantic vector corresponding to the target sentence", it can further include:

[0144] taking the target sentence as a first reference sentence, executing the step of performing semantic analysis on the first reference sentence to obtain the node semantic vector corresponding to the first reference sentence, and obtaining a new vector index file;

[0145] Detecting the number of target sentences to be detected, and if the number of target sentences to be detected is less than a preset value, replacing the currently used vector index file with a new vector index file.

[0146] The updating process can be performed after detecting each pair of target sentences, but in order to ensure user experience, the updating of the vector index file can be performed when certain conditions are met, for example, when the number of detected target sentences meets a certain condition, or when a certain time interval is reached, the vector index file is updated based on the target sentences detected within the time interval, and the like. The skilled person can set this according to actual needs, and the embodiments of the present application do not limit this.

[0147] In some embodiments, the currently used vector index file refers to a vector index file that is currently providing general content detection services for users.

[0148] In other embodiments, the constructed vector index file supports instant updating, and at this time, the vector index file can be updated instantaneously based on the target sentence. That is, after the step of "performing semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence", the following steps are further included:

[0149] The target sentence is used as a new first reference sentence, and semantic analysis is performed on the new first reference sentence to obtain a new node semantic vector corresponding to the new first reference sentence;

[0150] Based on a preset vector node calculation method, a new node corresponding to the new node semantic vector is calculated;

[0151] According to the distribution relationship between the new node and each node in the vector index, the new node is connected with the existing nodes to obtain a new node distribution relationship;

[0152] Based on the new node semantic vector and the new node distribution relationship, the vector index file is updated.

[0153] The vector index file can be constructed according to a navigable small world (NSW) algorithm, a hierarchical navigable small world (HNSW) algorithm, or the like.

[0154] 203、determining that the reference semantic vectors with a similarity greater than a preset similarity threshold are target reference semantic vectors, and obtaining the number of reference semantic vectors and the number of target reference semantic vectors.

[0155] The preset similarity threshold can be set by the skilled person according to actual use, for example, a similarity greater than 90%, and the like, and the embodiments of the present application do not limit this.

[0156] In some embodiments, as shown in Figure 4 The reference semantic vector can be obtained from a semantic library, in which node semantic vectors and a vector index file can be pre-stored. When general content detection is needed, the reference semantic vector can be determined from the node semantic vectors of the semantic library based on the target semantic vector and the vector index file.

[0157] 204. If the ratio between the number of target reference semantic vectors and the number of reference semantic vectors is greater than a preset value, it is determined that the target sentence belongs to a general sentence.

[0158] In some embodiments, the user-generated content only includes one sentence. When the target sentence is determined to be a general sentence, the user-generated content can be determined to be general content.

[0159] In other embodiments, the user-generated content can include at least two sentences. At this time, other sentences in the user-generated content also need to be determined as general sentences, i.e., the user-generated content includes at least one sentence.

[0160] After step 204, it further includes:

[0161] Another sentence is determined from the user interaction content as a target sentence to be detected, and the step of performing semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence is returned until all sentences in the user-generated content participate in general content detection.

[0162] The ratio between the number of target reference semantic vectors corresponding to each sentence in the user interaction content and the number of reference semantic vectors is recorded.

[0163] According to the ratio corresponding to each sentence, the general probability corresponding to the user-generated content is calculated. If the general probability is greater than a preset probability, the user-generated content is determined to be general content.

[0164] In some examples, each sentence in the user-generated content can be assigned a different weight, for example, according to the proportion of the number of words in each sentence in the user-generated content. The weight corresponding to each sentence is set. According to the weight of each sentence and the ratio corresponding to each sentence, the general probability corresponding to the user-generated content is calculated.

[0165] In one example of some examples, each sentence in the user-generated content can be assigned a different general score according to the ratio between the number of target reference semantic vectors corresponding to each sentence in the user interaction content and the number of reference semantic vectors. The score of each sentence and the weight are used to determine the score of the user-generated content as the general probability corresponding to the user-generated content.

[0166] In some examples, the number of target sentences belonging to the general sentences can be calculated according to a ratio between the number of target reference semantic vectors corresponding to each sentence in the user interaction content and the number of reference semantic vectors.

[0167] The ratio of the general sentences between the number of target sentences belonging to the general sentences and the number of all sentences in the user production content is calculated again, and if the ratio of the general sentences is greater than a preset ratio, the user production content is determined as the general content.

[0168] As shown in Figure 8 and Figure 9 , the embodiment of the present application shows the result example of the general content detection. For example Figure 8 , the score of each target sentence is calculated in the user production content, and whether the sentence is a catch-all sentence (general sentence) is determined according to the score of each sentence. Although Figure 8 , the user production content has both catch-all sentences and non-catch-all sentences, but the user production content as a whole can be determined as non-catch-all content according to the score and weight of each sentence and other parameters.

[0169] After determining that the user production content is general content, a technical person can take a sinking or non-displaying processing strategy. According to the sinking processing strategy, the user production content belonging to the general content can be displayed at the back of the terminal by sinking operation.

[0170] Alternatively, according to the non-displaying processing strategy, the user production content belonging to the general content can be not displayed on the terminal. For example, as shown in Figure 10 “4 user production contents are folded” in Figure 10 , the user can click Why to understand the reason why the user production content is folded. The specific processing strategy can be determined by the technical person, and the embodiment of the present application does not limit it.

[0171] In some embodiments, the technical person can also apply the detected general content in some scenarios, for example, in the game scenario, for the virtual characters that do not affect the game progress, the general content can be set for them to communicate with the game user. For example, in the virtual scene of the medical hall, the virtual characters that do not promote the game progress can be set with general content such as drinking more hot water to improve the user's interactive interest.

[0172] As can be seen from the above, the embodiment of the present application can filter out the content with low quality for content reading users from the user production content, improve the auditing efficiency of the user production content on the basis of ensuring the auditing accuracy.

[0173] According to the method described in the foregoing embodiments, the following will be further described in detail by way of examples.

[0174] In this embodiment, the system of Figure 1 will be described.

[0175] As shown in Figure 11 , the general content detection method of this embodiment can have the following specific process:

[0176] 301. The server acquires reference user production content, and constructs a semantic vector library based on the reference user production content.

[0177] The reference user production content can be acquired from other devices, such as other servers or terminals, or can also be acquired from the device (i.e., the device where the reference user production content is located).

[0178] It should be noted that when acquiring reference user production content from different devices, the reference user production content can be acquired simultaneously or separately, and the order of acquisition can be irrelevant when acquired separately.

[0179] In some examples, the semantic vector library can store node semantic vectors and vector index files. In other examples, the node semantic vectors and the vector index files can also be stored separately, and the embodiments of the present application do not limit this.

[0180] 302. The terminal receives user input user production content, and sends the user production content to the server.

[0181] The received user production content can be directly sent to the server by multiple terminals, or the user production content received by each terminal can be sent to a cloud server, and when the server needs to perform general content detection, the user production content to be detected can be acquired from the cloud server.

[0182] 303. The server acquires user production content, and pre-processes the user production content to obtain target sentences to be detected.

[0183] The server can pre-process the acquired user production content immediately, or can store the acquired user production content for a period of time and then pre-process the content that has been stored for a period of time. That is, after receiving the user production content, the server can store the user production content in a database, and when a preset general content preprocessing time (such as 01:00 every morning) arrives, the server can acquire the unprocessed user production content from the database for preprocessing.

[0184] In other examples, the pre-processing process can be performed on the terminal, and the embodiments of the present application do not repeat the description.

[0185] 304. The server performs semantic analysis on the target statement to be detected to obtain a target semantic vector corresponding to the target statement, wherein the target statement is from the user-generated content.

[0186] In some embodiments, the target semantic vector corresponding to the target statement can be obtained by obtaining a word vector corresponding to the target statement and performing calculation on the word vector. That is, step 304 can include:

[0187] performing word segmentation on the target statement to obtain words in the target statement;

[0188] mapping each word to a word vector corresponding to the word based on a preset vector mapping relationship;

[0189] performing calculation based on each word vector to obtain a target semantic vector corresponding to the target statement.

[0190] When mapping the word to the word vector, a semantic representation model such as a Word2vec model, a continuous bag of words (CBOW) model, or the like can also be used.

[0191] 305. The server determines the similarity between the target semantic vector and a reference semantic vector, wherein the reference semantic vector is a semantic vector corresponding to a reference statement, and the reference statement is from the reference user-generated content.

[0192] In some embodiments, the similarity between the target semantic vector and the reference semantic vector can be the Euclidean distance, Manhattan distance, cosine of the included angle, Pearson correlation coefficient, or the like between the target semantic vector and each reference semantic vector, and the calculated value is taken as the similarity between the target semantic vector and the reference semantic vector.

[0193] In other embodiments, to improve the accuracy of semantic relationship analysis, a semantic relationship representation model such as a bidirectional encoder representation from transformers (BERT) model, a Word2vec model, or the like can also be used for analysis.

[0194] 306. The server determines the reference semantic vector with a similarity greater than a preset similarity threshold as a target reference semantic vector, and obtains the number of reference semantic vectors and the number of target reference semantic vectors.

[0195] The preset similarity threshold can be set by the technician according to actual use, for example, greater than 95%, and the like, and the embodiments of the present application do not limit this.

[0196] 307、the server determines that the target sentence belongs to the general sentence if a ratio between a number of target reference semantic vectors and a number of reference semantic vectors is greater than a preset value.

[0197] The preset value can be 3 / 4, etc., for example, the number of reference semantics is 4, and the similarity between 3 reference semantic vectors and the target semantic vector is greater than the preset threshold value, and the 3 reference semantic vectors are taken as the target reference semantic vector.

[0198] 308、if the user production content includes at least two sentences, the server determines that the target sentence belongs to the general sentence, and then determines another sentence from the user interaction content as a target sentence to be detected, and returns to the step of performing semantic analysis on the target sentence to be detected to obtain the target semantic vector corresponding to the target sentence until all sentences in the user production content participate in the general content detection.

[0199] In some embodiments, the user production content only includes one sentence, and when the target sentence is determined to be a general sentence, the user production content can be determined to be general content.

[0200] In some embodiments, the user production content only includes one sentence, and when the target sentence is determined to be a general sentence, the user production content can be determined to be general content.

[0201] 309、the server calculates a general sentence ratio between a number of target sentences belonging to the general sentence and a number of all sentences in the user production content, and determines that the user production content is general content if the general sentence ratio is greater than a preset ratio.

[0202] In some embodiments, the user production content only includes one sentence, and when the target sentence is determined to be a general sentence, the user production content can be determined to be general content.

[0203] Alternatively, the user production content can be displayed according to the display processing strategy, so that the user production content belonging to the general content is not displayed on the terminal. The specific processing strategy can be determined by the technical personnel, and the embodiments of the present application do not limit this.

[0204] As can be seen from the above, the embodiments of the present application can screen out the content with low quality for the content reading user from the user production content, and improve the auditing efficiency of the user production content on the basis of ensuring the auditing accuracy.

[0205] In order to better implement the above method, correspondingly, the embodiments of the present application also provide a general content detection device.

[0206] Reference Figure 12The general content detection device comprises:

[0207] The semantic analysis unit 601 is configured to perform semantic analysis on the target statement to be detected to obtain a target semantic vector corresponding to the target statement, wherein the target statement is from user-generated content;

[0208] The similarity calculation unit 602 is configured to determine the similarity between the target semantic vector and a reference semantic vector, wherein the reference semantic vector is a semantic vector corresponding to a reference statement, and the reference statement is from reference user-generated content;

[0209] The target vector determination unit 603 is configured to determine the reference semantic vector with a similarity greater than a preset similarity threshold as a target reference semantic vector, and obtain the number of reference semantic vectors and the number of target reference semantic vectors;

[0210] The general statement determination unit 604 is configured to determine that the target statement belongs to a general statement if the ratio between the number of target reference semantic vectors and the number of reference semantic vectors is greater than a preset value.

[0211] In an optional example, the similarity calculation unit 602 is configured to perform semantic relationship analysis on the target semantic vector and each reference semantic vector to obtain a semantic relationship vector corresponding to the target semantic vector and each reference semantic vector;

[0212] The similarity between the target semantic vector and each reference semantic vector is obtained by performing semantic similarity analysis on each semantic relationship vector.

[0213] In an optional example, as shown in Figure 13 The vector search unit 605 is further configured to obtain a pre-established vector index file, wherein the vector index file comprises nodes corresponding to each node semantic vector and a node distribution relationship indicating the distribution relationship of each node semantic vector;

[0214] The target semantic vector is represented as a target node in the vector index file based on the node creation manner in the vector index file;

[0215] According to the target node and the vector index file, the reference node matching the target node is determined;

[0216] Based on the corresponding relationship between the node semantic vector and the node, the reference semantic vector corresponding to the reference node is determined from the node semantic vector.

[0217] In an optional example, the vector search unit 605 further comprises an index construction unit 606 configured to perform semantic analysis on the reference statement to obtain a node semantic vector corresponding to the reference statement;

[0218] According to the distribution relationship of the node semantic vectors, a node corresponding to each node semantic vector is created, and the distribution relationship between the nodes is associated with the distribution relationship of the node semantic vectors.

[0219] Based on the nodes corresponding to the node semantic vectors, the corresponding relationship between the node semantic vectors and the nodes, and the distribution relationship between the nodes, a vector index file is constructed.

[0220] In an optional example, after the index construction unit 606, a first update unit 607 can also be included, which is configured to take the target sentence as a reference sentence, perform semantic analysis on the reference sentence to obtain the node semantic vector corresponding to the reference sentence, and obtain a new vector index file.

[0221] The number of target sentences to be detected is detected, and if the number of target sentences to be detected is less than a preset value, the currently used vector index file is replaced with the new vector index file.

[0222] In an optional example, as shown in Figure 14 After the index construction unit 606, a second update unit 608 can also be included, which is configured to take the target sentence as a new reference sentence, perform semantic analysis on the reference sentence to obtain a new node semantic vector corresponding to the new reference sentence.

[0223] Based on a preset vector node calculation method, a new node corresponding to the new node semantic vector is calculated.

[0224] According to the new node and the distribution relationship between the nodes in the vector index, the new node is inserted into the nodes to obtain a new node distribution relationship.

[0225] Based on the new node semantic vector and the new node distribution relationship, the vector index file is updated.

[0226] In an optional example, the semantic analysis unit 601 is configured to perform semantic relationship analysis on the target semantic vector and each reference semantic vector through a semantic relationship representation model.

[0227] Before the semantic analysis unit 601, a model training unit 609 is further included, which is configured to obtain sample target semantic vectors and sample reference semantic vectors, and actual similarity of the sample target semantic vectors and the sample reference semantic vectors.

[0228] Based on a preset vector connection mark, the sample target semantic vectors and the sample reference semantic vectors are spliced to obtain a spliced vector.

[0229] Through the semantic relationship representation model to be trained, semantic relationship analysis is performed on the spliced vector to obtain a sample semantic relationship vector corresponding to the sample target semantic vector and the sample reference semantic vector.

[0230] According to the sample semantic relation vector, semantic approximation degree analysis is performed to obtain a predicted similarity between the sample target semantic vector and the sample reference semantic vector;

[0231] According to the predicted similarity and the actual similarity, a loss of the semantic relation representation model to be trained is determined;

[0232] According to the loss, a model parameter of the semantic relation representation model to be trained is adjusted to obtain a trained semantic relation representation model.

[0233] In an optional example, the user-generated content includes at least one sentence, and the general content detection apparatus further includes a general content determination unit 610 configured to determine another sentence from the user interaction content as a target sentence to be detected, and return to perform the step of performing semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence until all the sentences in the user-generated content participate in the general content detection.

[0234] Record the ratio between the number of target reference semantic vectors corresponding to each sentence in the user interaction content and the number of reference semantic vectors.

[0235] According to the ratio corresponding to each sentence, a general probability corresponding to the user-generated content is calculated, and if the general probability is greater than a preset probability, the user-generated content is determined as general content.

[0236] As can be seen from the above, the general content detection apparatus of the embodiment of the present application can filter out the content of low quality for content reading users from the user-generated content, thereby improving the auditing efficiency of the user-generated content on the basis of ensuring the auditing accuracy.

[0237] In addition, the embodiment of the present application further provides an electronic device, which can be a terminal or a server, etc. Figure 15 As shown in FIG. 10, a structure schematic diagram of the electronic device related to the embodiment of the present application is shown, and specifically:

[0238] The electronic device can include a radio frequency (RF, Radio Frequency) circuit 901, a memory 902 including one or more computer readable storage media, an input unit 903, a display unit 904, a sensor 905, an audio circuit 906, a wireless fidelity (WiFi, Wireless Fidelity) module 907, a processor 908 including one or more processing cores, and a power supply 909, etc. Those skilled in the art can understand that the terminal structure shown in FIG. 10 does not constitute a limitation on the electronic device, and can include more or fewer components than the diagram, or combine certain components, or different component arrangements. Among them: Figure 15 ​

[0239] The RF circuit 901 can be used for receiving and sending signals in the process of information or communication, in particular, receiving the downlink information from the base station and delivering it to the processor 908 for processing; in addition, sending the uplink data to the base station. Generally, the RF circuit 901 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, etc. In addition, the RF circuit 901 can also communicate with the network and other devices through wireless communication. The wireless communication can use any communication standards or protocols, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0240] The memory 902 can be used to store software programs and modules, and the processor 908 can execute various function applications and data processing by running the software programs and modules stored in the memory 902. The memory 902 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 902 can include a high-speed random access memory, and can also include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 902 can also include a memory controller to provide access for the processor 908 and the input unit 903 to the memory 902.

[0241] Input unit 903 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in one embodiment, input unit 903 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, can collect user touch operations on or near it (e.g., user operations using fingers, styluses, or any suitable object or accessory on or near the touch-sensitive surface) and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 908, and can receive and execute commands from the processor 908. Furthermore, various types of touch-sensitive surfaces, such as resistive, capacitive, infrared, and surface acoustic wave, can be used. In addition to the touch-sensitive surface, input unit 903 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0242] Display unit 904 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of electronic devices. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Display unit 904 may include a display panel, optionally configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar form. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to processor 908 to determine the type of touch event. Subsequently, processor 908 provides corresponding visual output on the display panel according to the type of touch event. Although in Figure 6 In this context, the touch-sensitive surface and the display panel are two separate components for implementing input and output functions. However, in some embodiments, the touch-sensitive surface and the display panel can be integrated to achieve both input and output functions.

[0243] The electronic device can further include at least one sensor 905, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor can include an ambient light sensor and a proximity sensor, where the ambient light sensor can adjust the brightness of the display panel according to the brightness of ambient light, and the proximity sensor can turn off the display panel and / or backlight when the electronic device is moved to the ear. As one of the motion sensors, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, it can detect the magnitude and direction of gravity, which can be used for identifying the posture of the mobile phone (such as switching between landscape and portrait, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), and the like. As for other sensors that the electronic device can also be configured, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, they will not be described here.

[0244] The audio circuit 906, the speaker, and the microphone can provide an audio interface between the user and the electronic device. The audio circuit 906 can convert the received audio data into an electrical signal and transmit it to the speaker, which converts the electrical signal into a sound signal and outputs it. On the other hand, the microphone collects a sound signal and converts it into an electrical signal, which is received by the audio circuit 906 and converted into audio data. The audio data is then output to the processor 908 for processing, and then transmitted to another electronic device via the RF circuit 901, or output to the memory 902 for further processing. The audio circuit 906 can also include a jack for connecting external earphones to the electronic device.

[0245] WiFi is a short-range wireless transmission technology. The WiFi module 907 can help the user to send and receive emails, browse web pages, and access streaming media, and it provides the user with wireless broadband Internet access. Although Figure 15 The WiFi module 907 is shown, but it is understood that it does not belong to the essential components of the electronic device, and can be omitted as needed without changing the essence of the application.

[0246] The processor 908 is the control center of the electronic device, which connects all parts of the mobile phone through various interfaces and lines, executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 902 and calling data stored in the memory 902, thereby overall detecting the mobile phone. Optionally, the processor 908 can include one or more processing cores; preferably, the processor 908 can integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 908.

[0247] The electronic device can also include a power source 909, such as a battery, for powering the various components of the device. In preferred embodiments, power source 909 is preferably coupled to power management system, which is in turn coupled to processor 908, so as to manage charging, discharging, and power consumption management, etc. Power source 909 can also include one or more DC or AC power sources, recharging systems, power failure detection circuitry, power conversion or inverter circuitry, power status indicator lights, and any other components.

[0248] Although not shown, the electronic device can also include a camera, a Bluetooth module, etc., which are not described herein. In the present embodiment, processor 908 loads one or more application programs into memory 902 and executes the application programs stored in memory 902 in accordance with the following instructions, thereby implementing various functions, such as:

[0249] performing semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence, wherein the target sentence is from user-generated content;

[0250] determining a similarity between the target semantic vector and a reference semantic vector, wherein the reference semantic vector is a semantic vector corresponding to a reference sentence, and the reference sentence is from reference user-generated content;

[0251] determining that a reference semantic vector with a similarity greater than a preset similarity threshold is a target reference semantic vector, and obtaining a number of reference semantic vectors and a number of target reference semantic vectors;

[0252] if a ratio between the number of target reference semantic vectors and the number of reference semantic vectors is greater than a preset value, determining that the target sentence is a general-purpose sentence.

[0253] Those of ordinary skill in the art can understand that all or part of the steps of the various methods of the above embodiments can be completed by instructions or by controlling relevant hardware by the instructions, which can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0254] To this end, an embodiment of the present application provides a storage medium having a plurality of instructions stored therein, which can be loaded by a processor to execute the steps of any one of the general-purpose content detection methods provided by the embodiments of the present application. For example, the instructions can execute the following steps:

[0255] performing semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence, wherein the target sentence is from user-generated content;

[0256] determine a similarity between the target semantic vector and a reference semantic vector, wherein the reference semantic vector is a semantic vector corresponding to a reference sentence from reference user generated content;

[0257] determine a reference semantic vector with a similarity greater than a preset similarity threshold as a target reference semantic vector, and obtain a quantity of reference semantic vectors and a quantity of target reference semantic vectors;

[0258] if a ratio between the quantity of target reference semantic vectors and the quantity of reference semantic vectors is greater than a preset value, determine that the target sentence belongs to a general sentence.

[0259] The specific implementation of each operation can be referred to the foregoing embodiments, which will not be repeated here.

[0260] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0261] Due to the instructions stored in the storage medium, the steps in any general content detection method provided by the embodiments of the present application can be performed, and thus the beneficial effects of any general content detection method provided by the embodiments of the present application can be achieved. Details are described in the foregoing embodiments, which will not be repeated here.

[0262] According to an aspect of the present application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the electronic device to perform the method provided in various optional implementation manners in the foregoing embodiments.

[0263] The above describes in detail a general content detection method, device, electronic device, and storage medium provided by the embodiments of the present application. The principle and implementation manner of the present application are described by applying specific examples. The above embodiment descriptions are only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, the specific implementation manner and application range can be changed according to the idea of the present application. The above description should not be understood as a limitation of the present application.

Claims

1. A general content detection method, characterized by, The method comprises the following steps: performing semantic analysis on a target sentence to be detected to obtain a target semantic vector corresponding to the target sentence, wherein the target sentence is from user-generated content; determining the similarity between the target semantic vector and a reference semantic vector, wherein the reference semantic vector is a semantic vector corresponding to a reference sentence, and the reference sentence is from reference user-generated content; determining a reference semantic vector with a similarity greater than a preset similarity threshold as a target reference semantic vector, and obtaining the number of reference semantic vectors and the number of target reference semantic vectors; if the ratio between the number of target reference semantic vectors and the number of reference semantic vectors is greater than a preset value, determining that the target sentence belongs to a general sentence.

2. The general content detection method according to claim 1, characterized in that, The method comprises the following steps: performing semantic relationship analysis on the target semantic vector and each reference semantic vector to obtain a semantic relationship vector corresponding to the target semantic vector and each reference semantic vector; performing semantic similarity analysis on each semantic relationship vector to obtain the similarity between the target semantic vector and each reference semantic vector.

3. The universal content detection method of claim 1, wherein, The reference semantic vector is from a node semantic vector; Before determining the similarity between the target semantic vector and the reference semantic vector, the method further comprises the following steps: obtaining a pre-established vector index file, wherein the vector index file comprises nodes corresponding to each node semantic vector and a node distribution relationship indicating the distribution relationship of each node semantic vector; based on the node creation method in the vector index file, representing the target semantic vector as a target node in the vector index file; determining a reference node matching the target node according to the target node and the vector index file; based on the correspondence between the node semantic vector and the node, determining a reference semantic vector corresponding to the reference node from the node semantic vector.

4. The general content detection method according to claim 3, characterized in that, Before obtaining the pre-established vector index file, the method further comprises the following steps: performing semantic analysis on a first reference sentence to obtain a node semantic vector corresponding to the first reference sentence, wherein the first reference sentence is from reference user-generated content; creating nodes corresponding to each node semantic vector according to the distribution relationship of each node semantic vector, wherein the distribution relationship between the nodes is associated with the distribution relationship of the node semantic vector; based on the nodes corresponding to each node semantic vector, the correspondence between each node semantic vector and the node, and the distribution relationship between the nodes, constructing the vector index file.

5. The general content detection method of claim 4, wherein, After performing semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence, the method further comprises the following steps: taking the target sentence as a first reference sentence, performing the step of performing semantic analysis on the first reference sentence to obtain a node semantic vector corresponding to the first reference sentence, to obtain a new vector index file; detecting the number of target sentences to be detected, and if the number of target sentences to be detected is less than a preset value, replacing the currently used vector index file with the new vector index file.

6. The general content detection method of claim 4, wherein, The semantic analysis of the target sentence to be detected includes: The target sentence is taken as a new first reference sentence, and the new first reference sentence is subjected to semantic analysis to obtain a new node semantic vector corresponding to the new first reference sentence; Based on a preset vector node calculation method, a new node corresponding to the new node semantic vector is calculated; According to the distribution relationship between the new node and each node in the vector index, the new node is connected with the existing nodes to obtain a new node distribution relationship; Based on the new node semantic vector and the new node distribution relationship, the vector index file is updated.

7. The general content detection method of claim 2, wherein, The semantic relationship analysis of the target semantic vector and each reference semantic vector includes: The semantic relationship analysis of the target semantic vector and each reference semantic vector is performed through a semantic relationship representation model; Before the semantic relationship analysis of the target semantic vector and each reference semantic vector through the semantic relationship representation model, the following steps are further included: Sample target semantic vectors and sample reference semantic vectors are obtained, as well as actual similarity of the sample target semantic vectors and sample reference semantic vectors; Based on a preset vector splicing mark, the sample target semantic vectors and sample reference semantic vectors are subjected to vector splicing to obtain splicing vectors; The splicing vectors are subjected to semantic relationship analysis through a to-be-trained semantic relationship representation model to obtain sample semantic relationship vectors corresponding to the sample target semantic vectors and sample reference semantic vectors; According to the sample semantic relationship vectors, semantic similarity analysis is performed to obtain a predicted similarity between the sample target semantic vectors and sample reference semantic vectors; According to the predicted similarity and the actual similarity, a loss of the to-be-trained semantic relationship representation model is determined; According to the loss, model parameters of the to-be-trained semantic relationship representation model are adjusted to obtain a trained semantic relationship representation model.

8. The method of claim 1-7, wherein, The user production content includes at least one sentence; After determining that the target sentence belongs to a general sentence, the following steps are further included: Another sentence is determined from the user production content as a target sentence to be detected, and the step of performing semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence is returned until all sentences in the user production content participate in general content detection; The ratio between the number of target reference semantic vectors corresponding to each sentence in the user production content and the number of reference semantic vectors is recorded; According to the ratio of each sentence, a general probability corresponding to the user production content is calculated, and if the general probability is greater than a preset probability, the user production content is determined as general content.

9. A general content detection apparatus characterized by comprising: The semantic analysis unit is configured to perform semantic analysis on a target sentence to be detected to obtain a target semantic vector corresponding to the target sentence, wherein the target sentence is from user production content. ​ The similarity calculation unit is configured to determine a similarity between the target semantic vector and a reference semantic vector, where the reference semantic vector is a semantic vector corresponding to a reference sentence from reference user generated content; The target vector determination unit is configured to determine a reference semantic vector with a similarity greater than a preset similarity threshold as a target reference semantic vector, and to obtain a quantity of the reference semantic vectors and a quantity of the target reference semantic vectors; The general sentence determination unit is configured to determine that the target sentence belongs to a general sentence if a ratio between the quantity of the target reference semantic vectors and the quantity of the reference semantic vectors is greater than a preset value.

10. The universal content detection device of claim 9, wherein, The similarity calculation unit is configured to perform semantic relationship analysis on the target semantic vector and each reference semantic vector to obtain a semantic relationship vector corresponding to the target semantic vector and each reference semantic vector; The similarity calculation unit is configured to perform semantic relationship analysis on the target semantic vector and each reference semantic vector to obtain a semantic relationship vector corresponding to the target semantic vector and each reference semantic vector; 11. The universal content detection device of claim 9, wherein, The reference semantic vector is from a node semantic vector, and the device further includes a vector search unit configured to obtain a pre-established vector index file, where the vector index file includes nodes corresponding to each node semantic vector and a node distribution relationship indicating a distribution relationship of each node semantic vector; The target semantic vector is represented as a target node in the vector index file based on a node creation manner in the vector index file; The target node and the vector index file are used to determine a reference node matched with the target node; The reference semantic vector corresponding to the reference node is determined from the node semantic vector based on a corresponding relationship between the node semantic vector and the node.

12. The general content detection apparatus according to claim 11, wherein The device further includes an index construction unit configured to perform semantic analysis on a first reference sentence to obtain a node semantic vector corresponding to the first reference sentence; Nodes corresponding to each node semantic vector are created based on a distribution relationship of each node semantic vector, where a distribution relationship between the nodes is associated with the distribution relationship of the node semantic vector; The vector index file is constructed based on the nodes corresponding to each node semantic vector, the corresponding relationship between each node semantic vector and the nodes, and the distribution relationship between the nodes.

13. The general content detection apparatus according to claim 12, characterized by The device further includes a first update unit configured to use the target sentence as a first reference sentence to perform the semantic analysis on the first reference sentence to obtain the node semantic vector corresponding to the first reference sentence, and to obtain a new vector index file; The device further includes a second update unit configured to use the target sentence as a new first reference sentence to perform semantic analysis on the new first reference sentence to obtain a new node semantic vector corresponding to the new first reference sentence; 14. The universal content detection device of claim 11, wherein, A new node corresponding to the new node semantic vector is calculated based on a preset vector node calculation method. ​ According to a distribution relationship between the new node and each of the nodes in the vector index, the new node is connected with the existing nodes to obtain a new node distribution relationship; Based on the new node semantic vector and the new node distribution relationship, the vector index file is updated.

15. The universal content detection device of claim 10, wherein, The semantic analysis unit is configured to perform semantic relationship analysis on the target semantic vector and each of the reference semantic vectors by using a semantic relationship representation model. The device further includes a model training unit configured to obtain a sample target semantic vector and a sample reference semantic vector, and an actual similarity between the sample target semantic vector and the sample reference semantic vector. Based on a preset vector connection mark, the sample target semantic vector and the sample reference semantic vector are spliced to obtain a spliced vector. The spliced vector is subjected to semantic relationship analysis by using a to-be-trained semantic relationship representation model to obtain a sample semantic relationship vector corresponding to the sample target semantic vector and the sample reference semantic vector. According to the sample semantic relationship vector, semantic similarity analysis is performed to obtain a predicted similarity between the sample target semantic vector and the sample reference semantic vector. According to the predicted similarity and the actual similarity, a loss of the to-be-trained semantic relationship representation model is determined. According to the loss, a model parameter of the to-be-trained semantic relationship representation model is adjusted to obtain a trained semantic relationship representation model.

16. A generic content detection apparatus according to any one of claims 9 to 15, characterized in that, The user-generated content includes at least one sentence, and the general content detection device further includes a general content determination unit configured to determine another sentence from the user-generated content as a target sentence to be detected, and return to perform the step of performing semantic analysis on the target sentence to be detected to obtain a target semantic vector corresponding to the target sentence until all sentences in the user-generated content participate in general content detection. A ratio between a number of target reference semantic vectors corresponding to each of the sentences in the user-generated content and a number of reference semantic vectors is recorded. According to the ratio corresponding to each of the sentences, a general probability corresponding to the user-generated content is calculated, and if the general probability is greater than a preset probability, the user-generated content is determined as general content.

17. An electronic device, comprising: The memory stores an application program, and the processor is configured to run the application program in the memory to perform the operations in the general content detection method of any one of claims 1 to 8.

18. A storage medium, characterized by The storage medium stores a plurality of instructions, and the instructions are adapted to be loaded by the processor to perform the operations in the general content detection method of any one of claims 1 to 8.

19. A computer program product, characterised in that, The computer program product includes computer instructions stored in a computer readable storage medium; the processor of the electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the electronic device perform the operations in the general content detection method of any one of claims 1 to 8.

Images