A data screening method, device, equipment and storage medium

Abstract

The present invention discloses a data screening method, device, equipment and storage medium. The method comprises: obtaining text vocabulary to be screened; performing collision matching between the text vocabulary to be screened and a weighted dictionary library to screen out target text vocabulary with a correlation greater than a set threshold; inputting the target text vocabulary into a TextRank algorithm to obtain a skeleton sentence; inputting the text vocabulary to be screened and the skeleton sentence into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set comprises: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples and skeleton sentence scoring samples. The technical solution of the present invention can solve the problem that maintenance personnel cannot perform corresponding maintenance according to extracted emotional vocabulary and / or cannot give corresponding rewards according to extracted emotional vocabulary.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a data screening method, device, equipment and storage medium. Background Art

[0002] Sentiment data computation and analysis is a common application of natural language processing (NLP) methods. It analyzes, processes, summarizes, and maps texts containing sentimental words to some quantitative data indicators, which can be understood as text classification. Traditional sentiment analysis methods are dictionary-based methods and machine learning algorithms. The dictionary analysis algorithm first needs to segment the text into words and sentences, then match the word list data with the dictionary one by one, and record the scores of the matched sentiment words. Finally, the sum of the sentiment scores of all the word segments is counted to obtain the sentiment tendency of the text.

[0003] The vector space model is a document modeling method commonly used to process text mining. It simplifies the processing of text content into vector operations in the vector space. When documents are represented as vectors in the document space, the similarity between documents can be measured by calculating the similarity between vectors. Some of its implementation methods include:

[0004] 1) N-gram model: Based on a certain corpus, N-Gram can be used to predict or evaluate whether a sentence is reasonable.

[0005] 2) TF-IDF model: If a word appears frequently in a document but rarely appears in other articles, it is considered that the word has good category discrimination ability.

[0006] 3) Paragraph Vector model: It is actually an extension of word vector. They add / average the word vectors of the text.

[0007] Sentiment algorithm based on dictionary analysis: The traditional analysis algorithm based on sentiment dictionary is limited by the quality and coverage of sentiment dictionary. Because it is a simple simulation of the human brain in principle, the core model is based on dictionary and rules, that is, the sentiment dictionary is used as the main basis for judging the sentiment polarity of comments, and a model is used to evaluate existing words based on past experience. For example, happy and joyful are usually regarded as positive words, and hateful and terrible are regarded as negative words. The sentiment model of the sentence is predicted by the sentiment words appearing in the sentence. In this method, the construction of the sentiment dictionary and the quality maintenance of the judgment rules require a lot of manpower, including manual experience and advanced knowledge.

[0008] Sentiment algorithm based on machine learning: The core of sentiment analysis algorithm based on machine learning lies in model training, which depends on the quality of labeled data sets. High-quality data requires a lot of labor costs and high-quality feature construction and selection.

[0009] In addition, the sentiment words selected based on the above method are only words used to express sentiments, such as: very good performance, not easy to use, etc. The maintenance personnel cannot perform corresponding maintenance based on the above sentiment words, or cannot give corresponding rewards based on the above sentiment words. Summary of the invention

[0010] The embodiments of the present invention provide a data screening method, device, equipment and storage medium to solve the problem that maintenance personnel are unable to perform corresponding maintenance according to extracted emotional words and / or are unable to give corresponding rewards according to the extracted emotional words.

[0011] In a first aspect, an embodiment of the present invention provides a data screening method, comprising:

[0012] Get the text vocabulary to be filtered;

[0013] Perform collision matching between the text words to be screened and the weighted dictionary library to screen out target text words whose correlation is greater than a set threshold;

[0014] Inputting the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence;

[0015] The text vocabulary to be screened and the skeleton sentence are input into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence score samples.

[0016] Furthermore, the neural network model is iteratively trained through the target sample set, including:

[0017] Build a neural network model;

[0018] Inputting text vocabulary samples in the target sample set and skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain predicted skeleton sentence scores;

[0019] Training the parameters of the neural network model according to an objective function formed by the predicted skeleton sentence score and the skeleton sentence score sample;

[0020] Return to executing the operation of inputting the text vocabulary samples in the target sample set and the skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain the predicted skeleton sentence scores until the causal inference model is obtained.

[0021] Furthermore, after inputting the text vocabulary samples in the target sample set and the skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain the predicted skeleton sentence scores, the method further includes:

[0022] Obtain the target skeleton sentence whose skeleton sentence score is greater than the score threshold;

[0023] Query a keyword list according to the target skeleton sentence to obtain keywords corresponding to the target skeleton sentence;

[0024] Determine the target address according to the keywords corresponding to the target skeleton sentence;

[0025] The target skeleton statement is sent to the target address.

[0026] Furthermore, the target text vocabulary is input into the TextRank algorithm to obtain a skeleton language, including:

[0027] Taking the target sentences in the target text vocabulary as nodes, and establishing edges between the nodes according to the relationships between the target sentences;

[0028] According to the nodes and edges, a skeleton statement is obtained.

[0029] Further, after obtaining the target skeleton sentence whose skeleton sentence score is greater than the score threshold, the method further includes:

[0030] Performing word segmentation processing on the target skeleton sentence to obtain at least one target word;

[0031] If the target word is found in the weighted dictionary, the weight of the target word in the weighted dictionary is increased;

[0032] If the target word is not found in the weighted dictionary library, the target word is stored in the weighted dictionary library.

[0033] Furthermore, the text vocabulary to be filtered is obtained, including:

[0034] Get evaluation text data;

[0035] The text words to be screened are preliminarily screened out from the evaluation text data by using the TF-IDF keyword weight calculation algorithm.

[0036] In a second aspect, an embodiment of the present invention further provides a data screening device, the device comprising:

[0037] An acquisition module is used to obtain text words to be filtered;

[0038] A screening module is used to perform collision matching between the text words to be screened and the weighted dictionary library, and screen out target text words whose correlation is greater than a set threshold;

[0039] A first determination module is used to input the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence;

[0040] The second determination module is used to input the text vocabulary to be screened and the skeleton sentence into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence score samples.

[0041] Furthermore, the second determination module is specifically used for:

[0042] Build a neural network model;

[0043] Inputting text vocabulary samples in the target sample set and skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain predicted skeleton sentence scores;

[0044] Training the parameters of the neural network model according to an objective function formed by the predicted skeleton sentence score and the skeleton sentence score sample;

[0045] Return to executing the operation of inputting the text vocabulary samples in the target sample set and the skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain the predicted skeleton sentence scores until the causal inference model is obtained.

[0046] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a data screening method as described in any one of the embodiments of the present invention is implemented.

[0047] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data screening method as described in any one of the embodiments of the present invention.

[0048] The embodiment of the present invention obtains text vocabulary to be screened; performs collision matching on the text vocabulary to be screened and a weighted dictionary library to screen out target text vocabulary with a correlation greater than a set threshold; inputs the target text vocabulary into a TextRank algorithm to obtain a skeleton sentence; inputs the text vocabulary to be screened and the skeleton sentence into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence scoring samples, so as to solve the problem that maintenance personnel cannot perform corresponding maintenance according to the extracted emotional vocabulary, and / or cannot give corresponding rewards according to the extracted emotional vocabulary, and can finally obtain sentences with causal relationships through the scoring of skeleton sentences, and according to the causal relationships in the sentences, maintenance personnel perform corresponding maintenance or give corresponding rewards. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 is a flow chart of a data screening method in an embodiment of the present invention;

[0051] Figure 2 is a structural schematic diagram of a data screening device in an embodiment of the present invention;

[0052] Figure 3 is a schematic structural diagram of an electronic device in an embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of the structure of a computer-readable storage medium containing a computer program in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0055] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0056] The term “including” and its variations used in the present invention are open inclusions, that is, “including but not limited to.” The term “based on” means “based at least in part on.” The term “one embodiment” means “at least one embodiment.”

[0057] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0058] Figure 1 A flowchart of a data screening method provided by an embodiment of the present invention. This embodiment is applicable to data screening. The method can be executed by a data screening device in an embodiment of the present invention. The data screening device can be implemented in software and / or hardware. Figure 1 As shown, the method specifically comprises the following steps:

[0059] S110, obtaining text words to be filtered.

[0060] Specifically, the method of obtaining the text vocabulary to be screened may be: obtaining evaluation text data, and determining the evaluation text data as the text vocabulary to be screened; the method of obtaining the text vocabulary to be screened may also be: obtaining the evaluation text data, and based on the currently common TF-IDF weight calculation method, obtaining keywords and their prevalence through word frequency (IF) and inverse document frequency (IDF), and preliminarily filtering out the text vocabulary to be screened that can provide more information. For example, it may be that the evaluation text data is obtained, and based on the currently common TF-IDF weight calculation method, the text vocabulary to be screened whose TF-IDF value is greater than the set TF-IDF value threshold is obtained.

[0061] S120, performing collision matching on the text words to be screened and the weighted dictionary library, and screening out target text words whose correlation is greater than a set threshold.

[0062] Among them, the dictionary library is mainly to integrate the public sentiment word dictionary (hownet sentiment dictionary), combined with known user characteristics (such as: IT engineer), and weighted modeling of some words. When the sentiment sentence hits the dictionary through the judgment, the sentence is weighted, and the "contribution" of each word in the sentence is averaged and added with the category bias to obtain the best sentiment sentence skeleton. The words in the skeleton are entered into the tag dictionary library. After training with the data after manual correction in the early stage, the correct rate is about 83% in the actual test, which can be continuously increased with the increase of data sets and manual correction.

[0063] Specifically, the text vocabulary to be screened is collided and matched with the weighted dictionary library to screen out target text vocabulary whose correlation is greater than a set threshold. For example, evaluation text data can be obtained, and based on the currently common TF-IDF weight calculation method, text vocabulary to be screened whose TF-IDF value is greater than a set TF-IDF value threshold is obtained, and the text vocabulary to be screened is collided with the weighted Tag dictionary library to further screen out target text vocabulary.

[0064] S130, inputting the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence.

[0065] Specifically, the target text vocabulary is input into the TextRank algorithm to obtain a skeleton sentence. For example, the evaluation text data is obtained, and based on the currently common TF-IDF weight calculation method, the keywords and their prevalence are obtained through the word frequency (IF) and the inverse document frequency (IDF), and the text vocabulary to be screened (with a large TF-IDF value) that can provide more information is initially filtered out. The text vocabulary to be screened is collided with the weighted Tag dictionary to further filter out the key words; combined with the principle of the TextRank algorithm, the extracted words are formed into nodes, and links are established based on the association of the words. According to the number of connected nodes, an initial weight value is assigned to each node. Then the iteration begins to obtain multiple skeleton sentences.

[0066] S140, inputting the text vocabulary to be screened and the skeleton sentence into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence score samples.

[0067] Specifically, counterfactual reasoning is used to perform tendency modeling and causal inference when multiple related sentiment sentences are obtained without scenario testing. Some words in the sentence are selected as feature variables A, and a conceptual model is established based on variable A. Variable A is manipulated to determine whether it can cause a change in result B. If it meets the causal relationship, it is considered to be a valid related sentiment sentence, otherwise it does not meet the causal relationship. In this way, the tendency of the conclusion can be calculated.

[0068] Optionally, iteratively train the neural network model using the target sample set, including:

[0069] Build a neural network model;

[0070] Inputting text vocabulary samples in the target sample set and skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain predicted skeleton sentence scores;

[0071] Training the parameters of the neural network model according to an objective function formed by the predicted skeleton sentence score and the skeleton sentence score sample;

[0072] Return to executing the operation of inputting the text vocabulary samples in the target sample set and the skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain the predicted skeleton sentence scores until the causal inference model is obtained.

[0073] Among them, the target sample set can be obtained by: obtaining historical evaluation data, annotating the historical evaluation data, annotating the skeleton sentences therein, and scoring the skeleton sentences. For example, the score of the skeleton sentence with a causal relationship is 100 points, and the score of the skeleton sentence without a causal relationship is 0 points. The embodiment of the present invention does not limit this.

[0074] It should be noted that the embodiments of the present invention do not limit the specific form of the neural network model.

[0075] Optionally, after inputting the text vocabulary samples in the target sample set and the skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain the predicted skeleton sentence scores, the method further includes:

[0076] Obtain the target skeleton sentence whose skeleton sentence score is greater than the score threshold;

[0077] Query a keyword list according to the target skeleton sentence to obtain keywords corresponding to the target skeleton sentence;

[0078] Determine the target address according to the keywords corresponding to the target skeleton sentence;

[0079] The target skeleton statement is sent to the target address.

[0080] The scoring threshold may be preset, or the skeleton sentence score with the highest score may be determined as the skeleton sentence score that is greater than the scoring threshold, which is not limited in this embodiment of the present invention.

[0081] The keyword list is a pre-established list of keywords.

[0082] Specifically, the purpose of querying the keyword list according to the target skeleton sentence and obtaining the keyword corresponding to the target skeleton sentence is to classify the target skeleton sentence. For example, the keyword list may be queried according to the target skeleton sentence and the target skeleton sentence may be obtained to be a skeleton sentence of the praise category. Then, based on the target skeleton sentence being in the praise category, the target address is determined to be the terminal device address corresponding to the department that receives statistical praise.

[0083] Specifically, we improve the weighted tag dictionary and semantic judgment through deep learning. The output involves one item of negative reviews for login and one item of negative reviews for transfer. Submitting the analysis data and original data to the corresponding departments can effectively solve customer demands and optimize applications.

[0084] Optionally, the target text vocabulary is input into the TextRank algorithm to obtain a skeleton language, including:

[0085] Taking the target sentences in the target text vocabulary as nodes, and establishing edges between the nodes according to the relationships between the target sentences;

[0086] According to the nodes and edges, a skeleton statement is obtained.

[0087] Specifically, combined with the principle of TextRank algorithm, the extracted words are formed into nodes, and links are established based on the association of words. According to the number of connected nodes, an initial weight value is assigned to each node. Then, iteration begins to obtain the skeleton sentence.

[0088] Optionally, after obtaining the target skeleton sentence whose skeleton sentence score is greater than the score threshold, the method further includes:

[0089] Performing word segmentation processing on the target skeleton sentence to obtain at least one target word;

[0090] If the target word is found in the weighted dictionary, the weight of the target word in the weighted dictionary is increased;

[0091] If the target word is not found in the weighted dictionary library, the target word is stored in the weighted dictionary library.

[0092] Specifically, taking into account the noise problem of the Chinese semantic model, combined with deep learning, neural networks and word tags are constructed to analyze, process and extract subjective texts with emotional colors, and continuously learn and improve the dictionary library.

[0093] Optionally, obtain the text vocabulary to be filtered, including:

[0094] Get evaluation text data;

[0095] The text words to be screened are preliminarily screened out from the evaluation text data by using the TF-IDF keyword weight calculation algorithm.

[0096] Specifically, the keyword weight calculation method is as follows: the evaluation text data is obtained, and based on the currently common TF-IDF weight calculation method, the keywords and their prevalence are obtained through the term frequency (IF) and inverse document frequency (IDF), and the text words to be screened (large TF-IDF value) that can provide more information are preliminarily filtered out.

[0097] In a specific example, the evaluation text data is obtained, and based on the currently common TF-IDF weight calculation method, the keywords and their prevalence are obtained through the word frequency (IF) and inverse document frequency (IDF), and the text words to be screened (large TF-IDF value) that can provide more information are initially filtered out. The text words to be screened are collided with the weighted Tag dictionary to further filter out the key words; combined with the principle of the TextRank algorithm, the extracted words are formed into nodes, and links are established based on the association of the words. According to the number of connected nodes, an initial weight value is assigned to each node. Then iterate to obtain multiple skeleton sentences. Causal inference algorithm (Causal Inference) based on the causal relationship between everything. Counterfactual reasoning is used to perform tendency modeling and causal inference when multiple related emotional sentences are obtained without scenario testing. Some words in the sentence are selected as feature variables A, and a conceptual model is established based on variable A. Variable A is operated to determine whether it can cause a change in result B. If it meets the causal relationship, it is considered to be a valid related emotional sentence, otherwise it does not meet the requirements. In this way, the tendency of the conclusion can be calculated.

[0098] In another specific example, in the data processing stage, it is mainly divided into four steps: (1) First, through the TF-IDF weight calculation method, the keywords and their prevalence are obtained through word frequency and inverse document frequency, and the important words that can provide more information are initially filtered out; (2) The weighted Tag dictionary words are collided, and the key words with high PMI (Point Mutual Information, correlation) related to the weighted Tag dictionary words (synonyms, antonyms, related words) are obtained; (3) Combined with the principle of TextRank algorithm, the extracted words are formed into nodes, and links are established based on the association of words. According to the number of connected nodes, an initial weight value is assigned to each node. Then the iteration begins to obtain the skeleton sentence. (4) Use the causal algorithm to determine whether the associated skeleton sentences have a causal relationship, and filter out the skeleton sentences with a causal relationship.

[0099] How to judge whether a passage is emotional language and whether there must be reasons or events that lead to this conclusion. To draw conclusions from massive data, it is necessary to establish a data model to analyze, process and extract subjective texts and evaluate whether this part of the data meets the judgment of the data model.

[0100] There are mainly the following concentrated scenarios, and it is recommended to consider using deep learning and weighted tag dictionary library to verify valid words:

[0101] A) For example, we processed and analyzed the reviews of personal mobile banking App Store. For the massive amount of reviews, we filtered out effective and important words through keyword weight calculation and dictionary analysis algorithm, and used some preset weighted tag dictionary libraries, such as "transfer", "login", "white screen" and other keywords to filter out key data. Combined with the principle of TextRank algorithm, the extracted words are formed into nodes, and links are established to form skeleton sentences based on the association of words. For example, the actual data "The input method of the verification code on the login interface always jumps out of error. The account number on the transfer interface must be entered manually. Once the previous input is wrong, it cannot be directly modified and all the numbers in the back must be deleted and re-entered." The key data "login", "error", "always", "transfer", "don't understand", "difficult" are filtered out. After TextRank processing, it has multiple semantics such as "login always makes mistakes, transfer is difficult", "login is difficult, transfer is wrong", etc. Using the causal algorithm to determine whether the skeleton sentence has a causal relationship, we can get the complete sentiment sentence of "login always goes wrong" because "the input method of the verification code on the login interface is always wrong" and "transfer is difficult" because "the account number on the transfer interface must be entered manually". Through deep learning, we can improve the weighted tag dictionary and semantic judgment. The output involves one negative review for login and one negative review for transfer. Submitting the analysis data and original data to the corresponding department can effectively solve customer demands and optimize applications. Through continuous learning and improvement of the dictionary library. By maintaining feature objects and manual correction, the accuracy of the algorithm model can approach 1 infinitely.

[0102] B) Process and analyze personal highlights data. For outstanding achievements, it is necessary to filter out effective content and give certain rewards. The weighted tag dictionary may be more inclined to help. Through the three stages of Text Preprocessing, Causality Relation, and Machine Learning Algorithm, a conclusion is drawn.

[0103] The embodiment of the present invention proposes a sentiment data screening scheme based on machine learning. The scheme mainly combines a keyword weight calculation algorithm, a causal inference algorithm, a canonical analysis model and deep learning, and is adapted to various evaluation screening, event causality, and sentiment data analysis scenarios. The scheme proposes a sentence-level text skeleton extraction algorithm. Through the general TF-IDF weight calculation method, keywords and their prevalence are obtained through word frequency and inverse document frequency, and important words that can provide more information are initially filtered out; then combined with the principle of the TextRank algorithm, the extracted words are formed into nodes, and links are established based on the association of the words. According to the number of connected nodes, an initial weight value is assigned to each node. Then iterate to obtain a skeleton sentence. The causal assertion algorithm is introduced, and counterfactual reasoning is used to perform tendency modeling and causal inference when multiple skeleton sentences are obtained without scenario testing. Some words in the sentence are selected as feature variables A, a conceptual model is established based on variable A, and variable A is operated to determine whether it can cause a change in result B. If it meets the causal relationship, it is considered to be a valid skeleton sentence, otherwise it does not meet the requirements. Combining the dictionary analysis model and deep learning, when the emotional sentence hits the dictionary through judgment, the sentence is weighted, and the "contribution" of each word in the sentence is averaged and added with the category bias to obtain the best emotional sentence skeleton. The words in the skeleton are entered into the Tag dictionary library and the parameters are updated through neural network propagation. The model parameter fitting noise is increased, which has a negative impact on the prediction.

[0104] The embodiments of the present invention are suitable for various evaluation screening, event causality and sentiment data analysis scenarios.

[0105] It should be noted that since we need to avoid manual extraction and achieve accurate analysis of high-quality data, deep learning algorithms based on machine learning algorithms are the most suitable for data screening. Deep learning is suitable for text data processing and semantic understanding. By using deep learning abstract features, it can simulate the connection between words, and has local feature abstraction and memory functions, which has great advantages in sentiment classification. At present, the deep learning neural networks used in sentiment analysis include multi-layer neural networks, convolutional neural networks, and long-short memory models. The common point of these neural networks is that they have many parameters and need to be propagated backward to update the parameters. It is very similar to the traditional machine learning principle: the more restrictions are imposed on the parameters or models, the smaller the degree of freedom of the model, and the less likely it is to overfit. Conversely, the more model parameters there are, the more flexible the model is, and the easier it is to fit noise, which has a negative impact on the prediction. The optimal parameters (such as the number of layers of models and the number of nodes per layer) are usually selected through cross-validation technology.

[0106] The data screening method proposed in the embodiment of the present invention mainly implements the principle of filtering out target text words through keyword weight calculation and dictionary analysis algorithm, forming skeleton sentences based on the target text words, and using the causal algorithm to judge whether the skeleton sentence has a causal relationship, and obtain a complete emotional sentence of A causing B.

[0107] The technical solution of this embodiment is as follows: obtaining text vocabulary to be screened; performing collision matching between the text vocabulary to be screened and a weighted dictionary library to screen out target text vocabulary with a correlation greater than a set threshold; inputting the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence; inputting the text vocabulary to be screened and the skeleton sentence into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence scoring samples, so as to solve the problem that maintenance personnel cannot perform corresponding maintenance according to the extracted emotional vocabulary, and / or cannot give corresponding rewards according to the extracted emotional vocabulary, and can finally obtain sentences with causal relationships through the scoring of skeleton sentences, and according to the causal relationships in the sentences, maintenance personnel can perform corresponding maintenance or give corresponding rewards.

[0108] Figure 2 This is a schematic diagram of the structure of a data screening device provided by an embodiment of the present invention. This embodiment is applicable to data screening. The device can be implemented in software and / or hardware. The device can be integrated in any device that provides data screening function, such as Figure 2 As shown, the data screening device specifically includes: an acquisition module 210, a screening module 220, a first determination module 230 and a second determination module 240.

[0109] Among them, the acquisition module is used to obtain the text vocabulary to be screened;

[0110] A screening module is used to perform collision matching between the text words to be screened and the weighted dictionary library, and screen out target text words whose correlation is greater than a set threshold;

[0111] A first determination module is used to input the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence;

[0112] The second determination module is used to input the text vocabulary to be screened and the skeleton sentence into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence score samples.

[0113] Optionally, the second determining module is specifically used for:

[0114] Build a neural network model;

[0115] Inputting text vocabulary samples in the target sample set and skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain predicted skeleton sentence scores;

[0116] Training the parameters of the neural network model according to an objective function formed by the predicted skeleton sentence score and the skeleton sentence score sample;

[0117] Return to executing the operation of inputting the text vocabulary samples in the target sample set and the skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain the predicted skeleton sentence scores until the causal inference model is obtained.

[0118] The above-mentioned product can execute the method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0119] The technical solution of this embodiment is as follows: obtaining text vocabulary to be screened; performing collision matching between the text vocabulary to be screened and a weighted dictionary library to screen out target text vocabulary with a correlation greater than a set threshold; inputting the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence; inputting the text vocabulary to be screened and the skeleton sentence into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence scoring samples, so as to solve the problem that maintenance personnel cannot perform corresponding maintenance according to the extracted emotional vocabulary, and / or cannot give corresponding rewards according to the extracted emotional vocabulary, and can finally obtain sentences with causal relationships through the scoring of skeleton sentences, and according to the causal relationships in the sentences, maintenance personnel can perform corresponding maintenance or give corresponding rewards.

[0120] Figure 3 FIG. 4 is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. Figure 3 A block diagram of an exemplary electronic device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 3 The electronic device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0121] like Figure 3 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0122] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0123] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0124] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 3 not shown, usually called a "hard drive"). Although Figure 3 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (a compact disc-read only memory (CD-ROM), a digital video disc (DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0125] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0126] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 22. In addition, the electronic device 12 in this embodiment, the display 24 does not exist as an independent individual, but is embedded in the mirror surface. When the display surface of the display 24 is not displayed, the display surface of the display 24 and the mirror surface are visually integrated. In addition, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 through a bus 18. It should be understood that although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0127] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, for example, implementing the data screening method provided in the embodiment of the present invention:

[0128] Get the text vocabulary to be filtered;

[0129] Perform collision matching between the text words to be screened and the weighted dictionary library to screen out target text words whose correlation is greater than a set threshold;

[0130] Inputting the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence;

[0131] The text vocabulary to be screened and the skeleton sentence are input into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence score samples.

[0132] Figure 4 Schematic diagram of the structure of a computer-readable storage medium containing a computer program in an embodiment of the present invention. The embodiment of the present invention provides a computer-readable storage medium 61 on which a computer program 610 is stored. When the program is executed by one or more processors, the data screening method provided in all the invention embodiments of the present application is implemented:

[0133] Get the text vocabulary to be filtered;

[0134] Perform collision matching between the text words to be screened and the weighted dictionary library to screen out target text words whose correlation is greater than a set threshold;

[0135] Inputting the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence;

[0136] The text vocabulary to be screened and the skeleton sentence are input into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence score samples.

[0137] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media or any combination of the above two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0138] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0139] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0140] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0141] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0142] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0145] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data screening method, characterized in that: include: Get the text vocabulary to be filtered; Perform collision matching between the text words to be screened and the weighted dictionary library to screen out target text words whose correlation is greater than a set threshold; Inputting the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence; Inputting the to-be-screened text vocabulary and the skeleton sentence into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, wherein the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence score samples; After inputting the text vocabulary samples in the target sample set and the skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain the predicted skeleton sentence scores, the method further includes: Obtain the target skeleton sentence whose skeleton sentence score is greater than the score threshold; Query a keyword list according to the target skeleton sentence to obtain keywords corresponding to the target skeleton sentence; Determine the target address according to the keywords corresponding to the target skeleton sentence; Sending the target skeleton statement to the target address; The target text vocabulary is input into the TextRank algorithm to obtain a skeleton sentence, including: Taking the target sentences in the target text vocabulary as nodes, and establishing edges between the nodes according to the relationships between the target sentences; According to the nodes and edges, a skeleton statement is obtained.

2. The method according to claim 1, characterized in that The neural network model is iteratively trained through the target sample set, including: Build a neural network model; Inputting text vocabulary samples in the target sample set and skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain predicted skeleton sentence scores; Training the parameters of the neural network model according to an objective function formed by the predicted skeleton sentence score and the skeleton sentence score sample; Return to executing the operation of inputting the text vocabulary samples in the target sample set and the skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain the predicted skeleton sentence scores until the causal inference model is obtained.

3. The method according to claim 1, characterized in that After obtaining the target skeleton sentence whose score is greater than the score threshold, it also includes: Performing word segmentation processing on the target skeleton sentence to obtain at least one target word; If the target word is found in the weighted dictionary, the weight of the target word in the weighted dictionary is increased; If the target word is not found in the weighted dictionary library, the target word is stored in the weighted dictionary library.

4. The method according to claim 1, characterized in that: Get the text vocabulary to be filtered, including: Get evaluation text data; The text words to be screened are preliminarily screened out from the evaluation text data by using the TF-IDF keyword weight calculation algorithm.

5. A data screening device, characterized in that: include: An acquisition module is used to obtain text words to be filtered; A screening module is used to perform collision matching between the text words to be screened and the weighted dictionary library, and screen out target text words whose correlation is greater than a set threshold; A first determination module is used to input the target text vocabulary into the TextRank algorithm to obtain a skeleton sentence; A second determination module is used to input the to-be-screened text vocabulary and the skeleton sentence into a causal inference model to obtain a score for the skeleton sentence, wherein the causal inference model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: text vocabulary samples, skeleton sentences corresponding to the text vocabulary samples, and skeleton sentence score samples; Also includes: A target skeleton sentence acquisition module is used to obtain a target skeleton sentence whose skeleton sentence score is greater than a score threshold after inputting the text vocabulary samples and the skeleton sentences corresponding to the text vocabulary samples in the target sample set into the neural network model to obtain a predicted skeleton sentence score; A keyword determination module corresponding to a target skeleton sentence, used to query a keyword list according to the target skeleton sentence to obtain keywords corresponding to the target skeleton sentence; A target address determination module, used to determine the target address according to the keywords corresponding to the target skeleton sentence; A sending module, used for sending the target skeleton statement to the target address; The first determination module is specifically used for: Taking the target sentences in the target text vocabulary as nodes, and establishing edges between the nodes according to the relationships between the target sentences; According to the nodes and edges, a skeleton statement is obtained.

6. The device according to claim 5, characterized in that The second determination module is specifically used for: Build a neural network model; Inputting text vocabulary samples in the target sample set and skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain predicted skeleton sentence scores; Training the parameters of the neural network model according to an objective function formed by the predicted skeleton sentence score and the skeleton sentence score sample; Return to executing the operation of inputting the text vocabulary samples in the target sample set and the skeleton sentences corresponding to the text vocabulary samples into the neural network model to obtain the predicted skeleton sentence scores until the causal inference model is obtained.

7. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the processors are enabled to implement the method according to any one of claims 1 to 4.

8. A computer-readable storage medium containing a computer program, wherein the computer program is stored thereon, characterized in that: When the program is executed by one or more processors, the method according to any one of claims 1 to 4 is implemented.

Images