Data acquisition and processing system for user text emotional tendency analysis

Abstract

The invention relates to the technical field of data processing, in particular to a data acquisition and processing system for user text emotional tendency analysis, which comprises an acquisition module, a cache module, a storage module, a trigger module, a sensitive information screening module, an interference information screening module and a verification module. According to the method, interference information and sensitive information are screened out, the screened to-be-verified content is verified to determine whether processing for the session content is qualified or not, and when it is judged that processing for the session content is abnormal, a processing mode for an information screening module is determined in time, so that the reliability of data used for emotion analysis is improved; and the accuracy of emotional tendency analysis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data acquisition and processing system for analyzing user text sentiment tendency. Background Art

[0002] The rapid development of artificial intelligence (AI) technology has made natural language processing (NLP) one of the important fields of intelligent applications. As a part of NLP, emotion recognition aims to automatically detect and analyze the user's emotional state from text, which is of great significance for improving user experience and optimizing service processes.

[0003] In the prior art, emotion recognition usually relies on rule-based classification methods or traditional machine learning models, and existing emotion recognition solutions mainly focus on the output of emotion classification results.

[0004] Chinese Patent Publication No.: CN109840328B discloses a method for analyzing sentiment tendency of commodity review text by deep learning, which processes the text to obtain the simplest text, and divides the simplest text into a training set and a test set, obtains a word sequence according to the training set and sets a sentiment feature with a sentiment weight value, and adjusts the sentiment weight value to obtain a corrected sentiment weight value, obtains multiple phrase sequences according to different combinations of constituent words in the word sequence, calculates the corrected sentiment weight value in the phrase sequence to obtain the sentiment label of the text, and compares the sentiment label with the artificial sentiment evaluation to obtain the final sentiment weight value and the final corrected sentiment weight value, performs the above operation on all texts in the training set, and verifies the sentiment analysis of the test set based on the sentiment model obtained according to the final sentiment weight value and the final corrected sentiment weight value; it can be seen that the above technical solution has the following problems: directly performing unified sentiment analysis on the simplest text after processing, without considering the verification of the data used for sentiment analysis, affecting the reliability of the data used for sentiment analysis, and thus leading to the problem of reduced accuracy of sentiment tendency analysis. Summary of the invention

[0005] To this end, the present invention provides a data acquisition and processing system for user text sentiment tendency analysis, so as to overcome the problem that the prior art directly performs unified sentiment analysis on the simplest text after processing, does not consider verifying the data used for sentiment analysis, affects the reliability of the data used for sentiment analysis, and further leads to reduced accuracy of sentiment tendency analysis.

[0006] To achieve the above object, the present invention provides a data collection and processing system for analyzing user text sentiment tendency, comprising: A collection module, which is used to collect user session content; A cache module, connected to the acquisition module, for caching the session content of the user; A storage module connected to the cache module, for obtaining the session content cached by the cache module when the amount of stored data in the cache module reaches a critical storage value, and for sending a content clearing instruction to the cache module when the caching of the session content is completed; A trigger module, which is connected to the cache module, periodically issues a trigger instruction with a preset trigger duration as a period, and obtains all session contents in the cache module when issuing the trigger instruction; A sensitive information screening module, which is connected to the trigger module and is used to screen out sensitive information in the conversation content obtained by the trigger module; An interference information screening module, which is connected to the sensitive information screening module, is used to screen out interference information from the session information after sensitive information screening is completed, and record the output session content after interference information screening is completed as content to be verified; A verification module, which is connected to the interference information screening module, the sensitive information screening module and the trigger module respectively, and is used to determine whether the processing of the session content is qualified based on the data volume of the content to be verified and the data volume of the session content obtained by the trigger module, including: Determine whether the processing of the conversation content is qualified, output the content to be verified, and perform text sentiment tendency analysis on the content to be verified; It is determined that the processing of the session content is abnormal, and the processing method for the information screening module is determined based on the data volume of the screened sensitive information and the data volume of the screened interference information.

[0007] Furthermore, the sensitive information screening module is used to screen out sensitive information in the conversation content acquired by the trigger module, including: Select a conversation window containing a preset conversation, perform natural sentence analysis on the conversation in the conversation window, and filter out sensitive information identified by the analysis; The interference information screening module is used to screen out interference information from the session information after sensitive information screening is completed, and includes: Determine the session information after sensitive information screening is completed as desensitized session information; Divide the desensitized conversation information into keywords, compare each keyword with each screening word in the screening standard library, determine the keywords with similarity greater than the preset similarity as interference information, and screen them out; The similarity is the ratio of the number of characters in the longest continuous identical part between the keyword and the filter word to the total number of characters in the filter word.

[0008] Further, the verification module is used to determine the screening ratio based on the data volume of the content to be verified and the data volume of the session content obtained by the trigger module, and determine whether the processing of the session content is qualified based on the screening ratio, including: To determine the screening ratio, the difference between the data volume of the session content obtained by the trigger module and the data volume of the content to be verified is calculated, and the ratio of the difference to the data volume of the session content obtained by the trigger module is solved to obtain the screening ratio; If the screening ratio is less than or equal to the first preset screening ratio, it is determined that the processing of the conversation content is qualified; If the screening ratio is less than or equal to the second preset screening ratio and greater than the first preset screening ratio, determining whether the processing of the session content is qualified in combination with the data volume of the session content obtained by the trigger module; If the screening ratio is greater than the second preset screening ratio, it is determined that the processing of the session content is abnormal, and the processing method for the information screening module is determined based on the data volume of the screened sensitive information and the data volume of the screened interference information.

[0009] Furthermore, the verification module is used to determine whether the processing of the session content is qualified based on the data volume of the session content obtained by the trigger module, including: The data volume of the session content acquired by the trigger module is recorded as the trigger data volume; If the trigger data volume is less than or equal to the preset data volume, it is determined that the processing of the session content is qualified, and the preset trigger duration is adjusted to a corresponding value based on the trigger data volume and the historical data increment; If the trigger data volume is greater than the preset data volume, it is determined that the processing of the session content is abnormal, and the processing method for the information screening module is determined based on the data volume of the filtered sensitive information and the data volume of the filtered interference information.

[0010] Further, the verification module is used to adjust the preset trigger duration to a corresponding value based on the amount of trigger data, wherein: The increase range of the preset trigger duration determined based on the amount of trigger data is inversely proportional to the amount of trigger data.

[0011] Further, the verification module adjusts the preset trigger duration to a corresponding value based on the historical data increment under the condition that the preset trigger duration is adjusted based on the trigger data amount, wherein: The increase of the preset trigger duration determined based on the historical data increment is inversely proportional to the historical data increment; The verification module is used to determine the historical data increment, draw the trigger data volume time domain curve based on the historical trigger data volume obtained by the trigger module, solve the slope of the trigger data volume time domain curve at the current time node, and determine the slope as the historical data increment.

[0012] Furthermore, the verification module is used to determine a screening comparison parameter based on the data volume of the screened sensitive information and the data volume of the screened interference information, and determine a processing method for the information screening module based on the screening comparison parameter, including: The ratio of the amount of data of the filtered interference information to the amount of data of the filtered sensitive information is recorded as the filtering comparison parameter; If the screening comparison parameter is less than or equal to the preset screening comparison parameter, adjusting the session window to a corresponding value based on the screening comparison parameter; If the screening comparison parameter is greater than the preset screening comparison parameter, the interference coincidence parameter is calculated based on the obtained interference information of each screening, and the processing method for the interference information screening module is determined based on the interference coincidence parameter.

[0013] Furthermore, the verification module is used to determine a processing method for the interference information screening module based on the interference coincidence parameter, including: To determine the interference overlap parameter, obtain the filtered interference information, divide the interference information into several data groups so that the same interference information is classified into the same data group, count the data volume of the interference information of each data group, and record the data volume of the data group with the largest data volume as the overlap data volume; solve the ratio of the overlap data volume to the total data volume of the filtered interference information to obtain the interference overlap parameter; If the interference overlap parameter is less than or equal to the preset interference overlap parameter, adjusting the preset similarity to a corresponding value based on the screening ratio; If the interference overlap parameter is greater than the preset interference overlap parameter, the preset similarity is adjusted to a corresponding value based on the screening ratio, and the number of each screening word in the screening standard library is adjusted to a corresponding value based on the interference overlap parameter.

[0014] Furthermore, the verification module is used to adjust the preset similarity to a corresponding value based on the screening ratio, wherein: The increase in the preset similarity determined based on the screening ratio is proportional to the screening ratio; The verification module is used to adjust the number of each screening word in the screening standard library to a corresponding value based on the interference overlap parameter, wherein: The increase in the number of each screening word in the screening standard library determined based on the interference overlap parameter is proportional to the interference overlap parameter.

[0015] Further, the verification module is used to adjust the session window to a corresponding value based on the screening comparison parameter, wherein: The increase of the session window determined based on the screening comparison parameter is inversely proportional to the screening comparison parameter.

[0016] Compared with the prior art, the beneficial effect of the present invention lies in that the data used for sentiment analysis is screened, interference information and sensitive information are screened out, and the content to be verified that has been screened out is verified to determine whether the processing of the conversation content is qualified. When it is determined that the processing of the conversation content is abnormal, the processing method for the information screening module is determined to improve the reliability of the data used for sentiment analysis, thereby improving the accuracy of sentiment tendency analysis.

[0017] Further, it is determined whether the processing of the conversation content is qualified. When the screening ratio is greater than the second preset screening ratio, too much screened content results in too little data for sentiment tendency analysis, and the accuracy of sentiment tendency analysis cannot be guaranteed. At this time, the processing method for the information screening module is determined to increase the amount of data for sentiment tendency analysis; when the screening ratio is less than or equal to the second preset screening ratio and greater than the first preset screening ratio, the data volume of the conversation content obtained by the comprehensive trigger module determines whether the processing of the conversation content is qualified. When the trigger data volume is less than or equal to the preset data volume, the obtained original data is too low, resulting in the inability to meet the data requirements of sentiment tendency analysis after a large amount of sensitive information and interference information are screened out. At this time, the preset trigger duration is increased to increase the amount of original data, thereby further improving the efficiency of acquiring data for sentiment analysis.

[0018] Furthermore, the processing method for the information screening module is determined. When the screening comparison parameter is less than or equal to the preset screening comparison parameter, the amount of sensitive information screened out is too large, so the amount of data retained for use and sentiment analysis is low. At this time, the session window is adjusted to improve the recognition accuracy of sensitive information; when the screening comparison parameter is greater than the preset screening comparison parameter, the recognition accuracy of interference information is low, resulting in too much data of the screened interference information. At this time, the interference overlap parameter is obtained, and the interference overlap parameter represents the overlap of the identified interference information. In this case, when the interference overlap parameter is greater than the preset interference overlap parameter, since the overlap of the interference information screened out is very high at this time, in order to ensure the accurate screening of the interference information, while adjusting the preset similarity to improve the screening standard, the content of the screening standard library is increased to ensure the accuracy of the screening. While ensuring the effective screening of interference information, the content of the screening standard library is streamlined to improve the screening efficiency of the interference data, further improving the efficiency of obtaining data used for sentiment analysis, thereby improving the efficiency of sentiment tendency analysis and improving the accuracy of sentiment tendency analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A block diagram of a data collection and processing system for analyzing user text sentiment tendency according to an embodiment of the present invention; Figure 2A logic decision diagram for determining whether processing of session content is qualified based on the screening ratio by the verification module in an embodiment of the present invention; Figure 3 A logic decision diagram for determining whether processing of a session content is qualified based on the data volume of the session content acquired by the trigger module in an embodiment of the present invention by the verification module; Figure 4 This is a logic decision diagram for the verification module of an embodiment of the present invention to determine the processing method for the information screening module based on the screening comparison parameter. DETAILED DESCRIPTION

[0020] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0021] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0022] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0023] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] See also Figure 1 , Figure 2 , Figure 3 as well as Figure 4As shown, they are respectively a module block diagram of a data acquisition and processing system for analyzing user text sentiment tendency according to an embodiment of the present invention, a logic decision diagram for determining whether the processing of a session content is qualified based on the screening ratio by a verification module, a logic decision diagram for determining whether the processing of a session content is qualified based on the data volume of the session content acquired by a trigger module, and a logic decision diagram for determining a processing method for an information screening module based on a screening comparison parameter by a verification module; a data acquisition and processing system for analyzing user text sentiment tendency according to an embodiment of the present invention comprises: A collection module, which is used to collect user session content; A cache module, connected to the acquisition module, for caching the session content of the user; A storage module connected to the cache module, for obtaining the session content cached by the cache module when the amount of stored data in the cache module reaches a critical storage value, and for sending a content clearing instruction to the cache module when the caching of the session content is completed; A trigger module, which is connected to the cache module, periodically issues a trigger instruction with a preset trigger duration as a period, and obtains all session contents in the cache module when issuing the trigger instruction; A sensitive information screening module, which is connected to the trigger module and is used to screen out sensitive information in the conversation content obtained by the trigger module; An interference information screening module, which is connected to the sensitive information screening module, is used to screen out interference information from the session information after sensitive information screening is completed, and record the output session content after interference information screening is completed as content to be verified; A verification module, which is connected to the interference information screening module, the sensitive information screening module and the trigger module respectively, and is used to determine whether the processing of the session content is qualified based on the data volume of the content to be verified and the data volume of the session content obtained by the trigger module, including: Determine whether the processing of the conversation content is qualified, output the content to be verified, and perform text sentiment tendency analysis on the content to be verified; It is determined that the processing of the session content is abnormal, and the processing method for the information screening module is determined based on the data volume of the screened sensitive information and the data volume of the screened interference information.

[0025] Specifically, the verification module outputs the conversation content that is determined to be qualified for conversation content processing to perform text sentiment tendency analysis on the conversation content. This is a prior art and will not be described in detail.

[0026] Specifically, the data used for sentiment analysis is screened to remove interference information and sensitive information, and the content to be verified that has been screened is verified to determine whether the processing of the conversation content is qualified. When it is determined that the processing of the conversation content is abnormal, the processing method for the information screening module is determined to improve the reliability of the data used for sentiment analysis, thereby improving the accuracy of sentiment tendency analysis.

[0027] Specifically, the sensitive information screening module is used to screen out sensitive information in the conversation content obtained by the trigger module, including: Select a conversation window containing a preset conversation, perform natural sentence analysis on the conversation in the conversation window, and filter out sensitive information identified by the analysis; The interference information screening module is used to screen out interference information from the session information after sensitive information screening is completed, and includes: Determine the session information after sensitive information screening is completed as desensitized session information; Divide the desensitized conversation information into keywords, compare each keyword with each screening word in the screening standard library, determine the keywords with similarity greater than the preset similarity as interference information, and screen them out; The similarity is the ratio of the number of characters in the longest continuous identical part between the keyword and the filter word to the total number of characters in the filter word.

[0028] Specifically, sensitive information includes personal identity information, financial information, health information, trade secrets, passwords and access credentials, regulatory information for specific industries, personal privacy information, and national security-related information, which are not listed here.

[0029] Specifically, there is no limitation on the specific method of performing natural sentence analysis on the conversation in the conversation window to obtain sensitive information identified by analysis. The AI ​​NLP recognition model can be used to analyze the chat content in each conversation window in real time to identify sensitive information, or the conversation in the conversation window can be imported into the detection neural network, and the detection neural network outputs the content of the imported conversation identified as sensitive information, and the content identified as sensitive information is screened out. This is a prior art and will not be elaborated on.

[0030] Specifically, the screening standard library is a pre-established collection of a series of screening words, which will not be described in detail here.

[0031] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] It should be pointed out that the data in this embodiment are obtained by comprehensive analysis and evaluation of the historical detection data and corresponding historical detection results of the system of the present invention in the three months before this detection. Before this detection, the system of the present invention comprehensively determines the values ​​of various preset parameter standards for this detection based on whether the processing of the conversation content of 15,863 users detected cumulatively in the first three months is qualified, the screening of sensitive information, the screening of interference information, and the specific analysis results of each text processing in each cycle. It can be understood by those skilled in the art that the determination method of the system of the present invention for a single parameter mentioned above can be to select the value with the highest proportion as the preset standard parameter according to the data distribution, use weighted summation to use the obtained value as the preset standard parameter or other selection methods, as long as the system of the present invention can clearly define different specific situations in the single determination process through the obtained value.

[0033] Specifically, the verification module is used to determine the screening ratio based on the data volume of the content to be verified and the data volume of the session content obtained by the trigger module, and determine whether the processing of the session content is qualified based on the screening ratio, including: To determine the screening ratio, the difference between the data volume of the session content obtained by the trigger module and the data volume of the content to be verified is calculated, and the ratio of the difference to the data volume of the session content obtained by the trigger module is solved to obtain the screening ratio; If the screening ratio is less than or equal to the first preset screening ratio, it is determined that the processing of the conversation content is qualified; If the screening ratio is less than or equal to the second preset screening ratio and greater than the first preset screening ratio, determining whether the processing of the session content is qualified in combination with the data volume of the session content obtained by the trigger module; If the screening ratio is greater than the second preset screening ratio, it is determined that the processing of the session content is abnormal, and the processing method for the information screening module is determined based on the data volume of the screened sensitive information and the data volume of the screened interference information.

[0034] Specifically, the first preset screening ratio S1 is selected within the interval [0.05, 0.15], and the second preset screening ratio S2 is selected within the interval [0.2, 0.25]; Specifically, in this embodiment, 0.15 is preferably selected as the value of S1, and 0.25 is selected as the value of S2. It is verified in 15863 experiments that when 0.15 is selected as the value of S1 and 0.25 is selected as the value of S2, the best division of the processing of the session content can be achieved, and the division of the proportion of the screened content can be more clearly achieved.

[0035] Specifically, the verification module is used to determine whether the processing of the session content is qualified based on the data volume of the session content obtained by the trigger module, including: The data volume of the session content acquired by the trigger module is recorded as the trigger data volume; If the trigger data volume is less than or equal to the preset data volume, it is determined that the processing of the session content is qualified, and the preset trigger duration is adjusted to a corresponding value based on the trigger data volume and the historical data increment; If the trigger data volume is greater than the preset data volume, it is determined that the processing of the session content is abnormal, and the processing method for the information screening module is determined based on the data volume of the filtered sensitive information and the data volume of the filtered interference information.

[0036] Specifically, the preset data volume Y0 is selected in the interval [200, 300], and the unit is MB; Specifically, in this embodiment, 260 is preferably selected as the value of Y0. It is verified in 15863 experiments that when 260 is selected as the value of Y0, the specific situation of the data volume of the conversation content obtained by the trigger module can be divided, and the situation of the obtained original data can be determined more clearly.

[0037] Specifically, it is determined whether the processing of the conversation content is qualified. When the screening ratio is greater than the second preset screening ratio, too much screened content results in too low a data volume for sentiment tendency analysis, resulting in the inability to ensure the accuracy of sentiment tendency analysis. At this time, the processing method for the information screening module is determined to increase the data volume for sentiment tendency analysis; when the screening ratio is less than or equal to the second preset screening ratio and greater than the first preset screening ratio, the data volume of the conversation content obtained by the comprehensive trigger module determines whether the processing of the conversation content is qualified. When the trigger data volume is less than or equal to the preset data volume, the obtained original data is too low, resulting in the inability to meet the data requirements of sentiment tendency analysis after a large amount of sensitive information and interference information are screened out. At this time, the preset trigger duration is increased to increase the data volume of the original data, thereby further improving the efficiency of acquiring data for sentiment analysis.

[0038] Specifically, the verification module is used to adjust the preset trigger duration to a corresponding value based on the amount of trigger data, wherein: The increase range of the preset trigger duration determined based on the amount of trigger data is inversely proportional to the amount of trigger data.

[0039] In this embodiment, optionally, Comparing the trigger data volume with a first preset trigger comparison threshold and a second preset trigger comparison threshold; If the amount of trigger data is less than or equal to the first preset trigger comparison threshold, the preset trigger duration is adjusted to 1.21 times the current preset trigger duration; If the amount of trigger data is less than or equal to the second preset trigger comparison threshold and greater than the first preset trigger comparison threshold, the preset trigger duration is adjusted to 1.17 times the current preset trigger duration; If the amount of trigger data is greater than the second preset trigger comparison threshold, the preset trigger duration is adjusted to 1.11 times the current preset trigger duration; The first preset trigger comparison threshold is 0.5Y0, and the second preset trigger comparison threshold is 0.7Y0.

[0040] Specifically, the verification module adjusts the preset trigger duration to a corresponding value based on the historical data increment under the condition that the preset trigger duration is adjusted based on the trigger data amount, wherein: The increase of the preset trigger duration determined based on the historical data increment is inversely proportional to the historical data increment; The verification module is used to determine the historical data increment, draw the trigger data volume time domain curve based on the historical trigger data volume obtained by the trigger module, solve the slope of the trigger data volume time domain curve at the current time node, and determine the slope as the historical data increment.

[0041] In this embodiment, optionally, Comparing the historical data increment with the first preset increment and the second preset increment; If the historical data increment is less than or equal to the first preset increment, the preset trigger duration is adjusted to 1.23 times the current preset trigger duration; If the historical data increment is less than or equal to the second preset increment and greater than the first preset increment, the preset trigger duration is adjusted to 1.19 times the current preset trigger duration; If the historical data increment is greater than the second preset increment, the preset trigger duration is adjusted to 1.12 times the current preset trigger duration; The first preset increment is -1.5, and the second preset increment is -0.5.

[0042] Specifically, the verification module is used to determine the screening comparison parameter based on the data volume of the screened sensitive information and the data volume of the screened interference information, and determine the processing method for the information screening module based on the screening comparison parameter, including: The ratio of the amount of data of the filtered interference information to the amount of data of the filtered sensitive information is recorded as the filtering comparison parameter; If the screening comparison parameter is less than or equal to the preset screening comparison parameter, adjusting the session window to a corresponding value based on the screening comparison parameter; If the screening comparison parameter is greater than the preset screening comparison parameter, the interference coincidence parameter is calculated based on the obtained interference information of each screening, and the processing method for the interference information screening module is determined based on the interference coincidence parameter.

[0043] Specifically, the preset screening comparison parameter L0 is selected within the interval [0.82, 0.93].

[0044] Specifically, the verification module is used to determine a processing method for the interference information screening module based on the interference coincidence parameter, including: To determine the interference overlap parameter, obtain the filtered interference information, divide the interference information into several data groups so that the same interference information is classified into the same data group, count the data volume of the interference information of each data group, and record the data volume of the data group with the largest data volume as the overlap data volume; solve the ratio of the overlap data volume to the total data volume of the filtered interference information to obtain the interference overlap parameter; If the interference coincidence parameter is less than or equal to the preset interference coincidence parameter, adjusting the preset similarity to a corresponding value based on the screening ratio; If the interference overlap parameter is greater than the preset interference overlap parameter, the preset similarity is adjusted to a corresponding value based on the screening ratio, and the number of each screening word in the screening standard library is adjusted to a corresponding value based on the interference overlap parameter.

[0045] Specifically, the preset interference coincidence parameter G0 is selected within the interval [0.29, 0.35].

[0046] Specifically, there is no restriction on the method of obtaining new filter words in the screening standard library. Domain experts, business analysts or relevant staff can manually determine and add new filter words according to business changes and regulatory updates, or track new words and buzzwords on the Internet, social media and other channels in real time, and determine texts evaluated as interference information as filter words. This will not be elaborated here.

[0047] Specifically, the processing method for the information screening module is determined. When the screening comparison parameter is less than or equal to the preset screening comparison parameter, the amount of sensitive information screened out is too large, so the amount of data retained for sentiment tendency analysis is low. At this time, the session window is adjusted to improve the recognition accuracy of sensitive information; when the screening comparison parameter is greater than the preset screening comparison parameter, the recognition accuracy of interference information is low, resulting in too much data of the screened interference information. At this time, the interference overlap parameter is obtained, and the interference overlap parameter characterizes the overlap of the identified interference information. When the interference overlap parameter is greater than the preset interference overlap parameter, the overlap degree of the screened interference information is very high. Therefore, in order to ensure the accurate screening of interference information, while adjusting the preset similarity to improve the screening standard, the content of the screening standard library is increased to ensure the accuracy of the screening. While ensuring the effective screening of interference information, the content in the screening standard library is streamlined to improve the screening efficiency of interference data, further improving the efficiency of obtaining data for sentiment analysis, thereby improving the efficiency of sentiment tendency analysis.

[0048] Specifically, the verification module is used to adjust the preset similarity to a corresponding value based on the screening ratio, wherein: The increase in the preset similarity determined based on the screening ratio is proportional to the screening ratio; The verification module is used to adjust the number of each screening word in the screening standard library to a corresponding value based on the interference overlap parameter, wherein: The increase in the number of each screening word in the screening standard library determined based on the interference overlap parameter is proportional to the interference overlap parameter.

[0049] In this embodiment, optionally, Comparing the screening ratio with a first preset ratio comparison threshold and a second preset ratio comparison threshold; If the screening ratio is less than or equal to the first preset ratio comparison threshold, the preset similarity is adjusted to 1.11 times the initial preset similarity; If the screening ratio is less than or equal to the second preset ratio comparison threshold and greater than the first preset ratio comparison threshold, the preset similarity is adjusted to 1.18 times the initial preset similarity; If the screening ratio is greater than the second preset ratio comparison threshold, the preset similarity is adjusted to 1.26 times the initial preset similarity; The first preset ratio comparison threshold is 1.7S2, and the second preset ratio comparison threshold is 3S2.

[0050] In this embodiment, optionally, Comparing the interference coincidence parameter with a first preset interference comparison threshold and a second preset interference comparison threshold; If the interference overlap parameter is less than or equal to the first preset interference comparison threshold, the number of each screening word in the screening standard library is adjusted to 1.13 times the initial number; If the interference overlap parameter is less than or equal to the second preset interference comparison threshold and greater than the first preset interference comparison threshold, the number of each screening word in the screening standard library is adjusted to 1.23 times the initial number; If the interference overlap parameter is greater than the second preset interference comparison threshold, the number of each screening word in the screening standard library is adjusted to 1.33 times the initial number; The first preset interference comparison threshold is 0.8G0, and the second preset interference comparison threshold is 2.1G0.

[0051] Specifically, the verification module is used to adjust the session window to a corresponding value based on the screening comparison parameter, wherein: The increase of the session window determined based on the screening comparison parameter is inversely proportional to the screening comparison parameter.

[0052] In this embodiment, optionally, Comparing the screening comparison parameter with a first preset screening comparison threshold and a second preset screening comparison threshold; If the screening comparison parameter is less than or equal to the first preset screening comparison threshold, the number of preset bar conversations in the conversation window is adjusted to 1.28 times of the initial preset bar; If the screening comparison parameter is less than or equal to the second preset screening comparison threshold and greater than the first preset screening comparison threshold, the number of preset bar conversations in the conversation window is adjusted to 1.18 times the initial preset bar; If the screening comparison parameter is greater than the second preset screening comparison threshold, the number of preset bar conversations in the conversation window is adjusted to 1.11 times the initial preset bar; The first preset screening comparison threshold is 0.63L0, and the second preset screening comparison threshold is 0.84L0.

[0053] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data collection and processing system for user text sentiment analysis, which is used for analyzing and processing long texts in user text sentiment analysis, characterized in that: include: A collection module, which is used to collect user session content; A cache module, connected to the acquisition module, for caching the session content of the user; A storage module connected to the cache module, for obtaining the session content cached by the cache module when the amount of stored data in the cache module reaches a critical storage value, and for sending a content clearing instruction to the cache module when the caching of the session content is completed; A trigger module, which is connected to the cache module, periodically issues a trigger instruction with a preset trigger duration as a period, and obtains all session contents in the cache module when issuing the trigger instruction; A sensitive information screening module, which is connected to the trigger module and is used to screen out sensitive information in the conversation content obtained by the trigger module; An interference information screening module, which is connected to the sensitive information screening module, is used to screen out interference information from the session information after sensitive information screening is completed, and record the output session content after interference information screening is completed as content to be verified; A verification module, which is connected to the interference information screening module, the sensitive information screening module and the trigger module respectively, and is used to determine whether the processing of the session content is qualified based on the data volume of the content to be verified and the data volume of the session content obtained by the trigger module, including: Determine whether the processing of the conversation content is qualified, output the content to be verified, and perform text sentiment tendency analysis on the content to be verified; It is determined that the processing of the session content is abnormal, and the processing method for the information screening module is determined based on the data volume of the screened sensitive information and the data volume of the screened interference information.

2. The data collection and processing system for analyzing user text sentiment tendency according to claim 1 is characterized in that: The sensitive information screening module is used to screen out sensitive information in the session content obtained by the trigger module, including: Select a conversation window containing a preset conversation, perform natural sentence analysis on the conversation in the conversation window, and filter out sensitive information identified by the analysis; The interference information screening module is used to screen out interference information from the session information after sensitive information screening is completed, including: Determine the session information after sensitive information screening is completed as desensitized session information; Divide the desensitized conversation information into keywords, compare each keyword with each screening word in the screening standard library, determine the keywords with similarity greater than the preset similarity as interference information, and screen them out; The similarity is the ratio of the number of characters in the longest continuous identical part between the keyword and the filter word to the total number of characters in the filter word.

3. The data collection and processing system for analyzing user text sentiment tendency according to claim 2 is characterized in that: The verification module is used to determine the screening ratio based on the data volume of the content to be verified and the data volume of the session content obtained by the trigger module, and determine whether the processing of the session content is qualified based on the screening ratio, including: To determine the screening ratio, the difference between the data volume of the session content obtained by the trigger module and the data volume of the content to be verified is calculated, and the ratio of the difference to the data volume of the session content obtained by the trigger module is solved to obtain the screening ratio; If the screening ratio is less than or equal to the first preset screening ratio, it is determined that the processing of the conversation content is qualified; If the screening ratio is less than or equal to the second preset screening ratio and greater than the first preset screening ratio, determining whether the processing of the session content is qualified in combination with the data volume of the session content obtained by the trigger module; If the screening ratio is greater than the second preset screening ratio, it is determined that the processing of the session content is abnormal, and the processing method for the information screening module is determined based on the data volume of the screened sensitive information and the data volume of the screened interference information.

4. The data collection and processing system for analyzing user text sentiment tendency according to claim 3 is characterized in that: The verification module is used to determine whether the processing of the session content is qualified based on the data volume of the session content obtained by the trigger module, including: The data volume of the session content acquired by the trigger module is recorded as the trigger data volume; If the trigger data volume is less than or equal to the preset data volume, it is determined that the processing of the session content is qualified, and the preset trigger duration is adjusted to a corresponding value based on the trigger data volume and the historical data increment; If the trigger data volume is greater than the preset data volume, it is determined that the processing of the session content is abnormal, and the processing method for the information screening module is determined based on the data volume of the filtered sensitive information and the data volume of the filtered interference information.

5. The data collection and processing system for analyzing user text sentiment tendency according to claim 4 is characterized in that: The verification module is used to adjust the preset trigger duration to a corresponding value based on the amount of trigger data, wherein: The increase range of the preset trigger duration determined based on the amount of trigger data is inversely proportional to the amount of trigger data.

6. The data collection and processing system for analyzing user text sentiment tendency according to claim 5 is characterized in that: The verification module adjusts the preset trigger duration to a corresponding value based on the historical data increment under the condition that the preset trigger duration is adjusted based on the trigger data amount, wherein: The increase of the preset trigger duration determined based on the historical data increment is inversely proportional to the historical data increment; The verification module is used to determine the historical data increment, draw the trigger data volume time domain curve based on the historical trigger data volume obtained by the trigger module, solve the slope of the trigger data volume time domain curve at the current time node, and determine the slope as the historical data increment.

7. The data collection and processing system for analyzing user text sentiment tendency according to claim 6 is characterized in that: The verification module is used to determine the screening comparison parameter based on the data volume of the screened sensitive information and the data volume of the screened interference information, and determine the processing method for the information screening module based on the screening comparison parameter, including: The ratio of the amount of data of the filtered interference information to the amount of data of the filtered sensitive information is recorded as the filtering comparison parameter; If the screening comparison parameter is less than or equal to the preset screening comparison parameter, adjusting the session window to a corresponding value based on the screening comparison parameter; If the screening comparison parameter is greater than the preset screening comparison parameter, the interference coincidence parameter is calculated based on the obtained interference information of each screening, and the processing method for the interference information screening module is determined based on the interference coincidence parameter.

8. The data collection and processing system for analyzing user text sentiment tendency according to claim 7 is characterized in that: The verification module is used to determine a processing method for the interference information screening module based on the interference coincidence parameter, including: To determine the interference overlap parameter, obtain the filtered interference information, divide the interference information into several data groups so that the same interference information is classified into the same data group, count the data volume of the interference information of each data group, and record the data volume of the data group with the largest data volume as the overlap data volume; solve the ratio of the overlap data volume to the total data volume of the filtered interference information to obtain the interference overlap parameter; If the interference overlap parameter is less than or equal to the preset interference overlap parameter, adjusting the preset similarity to a corresponding value based on the screening ratio; If the interference overlap parameter is greater than the preset interference overlap parameter, the preset similarity is adjusted to a corresponding value based on the screening ratio, and the number of each screening word in the screening standard library is adjusted to a corresponding value based on the interference overlap parameter.

9. The data collection and processing system for analyzing user text sentiment tendency according to claim 8 is characterized in that: The verification module is used to adjust the preset similarity to a corresponding value based on the screening ratio, wherein: The increase in the preset similarity determined based on the screening ratio is proportional to the screening ratio; The verification module is used to adjust the number of each screening word in the screening standard library to a corresponding value based on the interference overlap parameter, wherein: The increase in the number of each screening word in the screening standard library determined based on the interference overlap parameter is proportional to the interference overlap parameter.

10. The data collection and processing system for analyzing user text sentiment tendency according to claim 9, characterized in that: The verification module is used to adjust the session window to a corresponding value based on the screening comparison parameter, wherein: The increase of the session window determined based on the screening comparison parameter is inversely proportional to the screening comparison parameter.

Images