A method for identifying harmful statements in the private chat function of a dating app

Abstract

This invention discloses a method for identifying harmful statements in the private chat function of a dating app. The method includes the following steps: establishing a list of emoticons; assigning a frequency probability value to each extended meaning of the same emoticon; replacing emoticons in the message with their corresponding codes; setting detection conditions; if a message does not meet the conditions, supplementing and combining the information; if the conditions are met, directly detecting the message; using a sliding window mechanism to detect the transformed sentences, determining whether the message is harmful after verification, and deciding whether to send the message based on its harmfulness; updating the list of high-risk emoticons based on customer feedback; and periodically updating the usage frequency of each extended meaning of the emoticon. The beneficial effects of this invention are: preventing malicious individuals from using combinations of emoticons or combinations of emoticons and text to convey information with harassing, fraudulent, or violent connotations.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, in particular to a harmful sentence recognition method for private chat function of a photo shooting software. BACKGROUND

[0002] The chat function is usually built-in in the photo shooting software, which is used for further communication between users. The chat function is an important tool for users to communicate the details of the photo shooting. Since the users of the photo shooting software are mostly young people, including models, makeup artists, photographers and other professional practitioners and other users with shooting needs, in addition to using the usual text expressions, emojis are often used to express emotions and information during the chat process. Emojis have the characteristics of being concise and having multiple meanings, and their bright colors and designs have memory points, which are deeply loved by young people and are often used in communication.

[0003] However, it is precisely because of the characteristics of emojis that some unscrupulous people often use these emojis, use combinations of emojis or combinations of emojis and text to convey information with meanings such as harassment, fraud, and violence. Some software with chat functions also have built-in harmful information detection systems, but these detection systems usually only target text and cannot detect potential bad information hints in emojis. Some people will use this way to bypass the platform's detection system and transmit bad information.

[0004] At the same time, the existing detection is usually performed when the send button is pressed, but this method has a problem. When the input content is prompted to contain harmful content / sensitive content / violation of community convention content, the user may not know where the sensitive content is, and thus cannot modify it.

[0005] Therefore, there is an urgent need for a harmful sentence recognition method for private chat function of a photo shooting software to solve the above problems. SUMMARY

[0006] The purpose of the present application is to provide a harmful sentence recognition method for private chat function of a photo shooting software to solve the problems raised in the background.

[0007] To solve the above technical problems, the present application provides the following technical solution, a harmful sentence recognition method for private chat function of a photo shooting software, comprising the following steps:

[0008] Step A: Establish an emoji list, and mark a usage frequency probability value for each connotative meaning of the same emoji;

[0009] Step B: Replace the emojis in the input message with corresponding codes;

[0010] Step C: Set up detection conditions, if the input message does not meet the conditions, the input message is supplemented and combined, and the input message that meets the conditions is directly detected;

[0011] Step D: Use the sliding window mechanism to detect the input message that has been transformed, and if there are illegal words in the input message after inspection, the user is reminded and the message is prevented from being sent.

[0012] Preferably, it also includes:

[0013] Step E: Update the high-risk emoji list according to customer feedback, and regularly update the usage frequency of each connotation of the emoji.

[0014] Preferably, the emoji in step A has a unique corresponding code, and the code corresponding to the unique emoji corresponds to multiple connotations.

[0015] Preferably, step C includes:

[0016] Step C1, set the detection condition to be no less than four bytes;

[0017] Step C2, if the input message does not meet the detection condition, the sent message and the input message will be combined until the length of the input message that does not pass the detection meets the detection condition;

[0018] Step C3, detect the input message that meets the condition.

[0019] Preferably, step D includes:

[0020] Step D1, set the sliding window to 6 code units, and set the step size to one code unit;

[0021] Step D2, when the input message exceeds the capacity size of the sliding window, the sliding window starts to slide and detect the input message, starting from the beginning of the input message, each time the window size is intercepted, and each time the step size is slid;

[0022] Step D3, each time the sliding window slides, a substring is generated in the string, if the substring contains an emoji, find the corresponding meaning group according to the emoji code, replace the emoji content with the meaning, and then perform semantic detection on the replaced message;

[0023] Step D4, if the statement is detected to contain harmful information, the message is prevented from being sent, and the customer is pointed out the illegal words in the input message.

[0024] Preferably, it also includes:

[0025] Step D5, repeat steps D1-D4, no harmful information detected, then allow the message to be sent out.

[0026] Preferably, the step D sliding window detection is performed simultaneously with the user input message, and the customer is reminded in real time.

[0027] Preferably, the step D3 further comprises:

[0028] When the meaning group replacement is performed, the initial meaning of the emoji is replaced in the sentence first, and then the extended meaning is used for replacement and semantic detection of the replaced message.

[0029] Preferably, the replacement order of the extended meaning is determined according to the use frequency probability value of the mark, and the extended meaning with high probability value is replaced first.

[0030] Compared with the prior art, the beneficial effects of the present application are:

[0031] 1、The present application replaces the emoji in the message with a unique code, thereby deforming the entire sentence into a string for processing; the sliding window mechanism is used to detect the deformed sentence, the emoji code in the sliding window is replaced with multiple extended meanings, and semantic detection is performed on each replaced extended meaning, thereby preventing malicious persons from using the combination of emojis or the combination of emojis and characters to transmit information with harassment, fraud, and violence.

[0032] 2、The present application detects the input message in real time through the sliding window mechanism, realizes input detection at the same time, and enables the user to know the illegal words in the input message in time and modify the illegal words in time.

[0033] 3、The present application determines the semantic replacement order of the emoji according to the use frequency probability value of the mark, and the extended meaning with high probability value is replaced first, thereby preventing malicious persons from using different extended meanings of the emoji to transmit information with harassment, fraud, and violence.

[0034] 4、The present application updates the high-risk list according to the user feedback, and the use frequency of each extended meaning of the emoji should also be regularly recalculated to update the use probability of each extended meaning of the emoji, so as to ensure that the latest use trend of the extended meaning is covered, and prevent malicious persons from using new extended meanings of the emoji to transmit information with harassment, fraud, and violence. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The flowchart of the harmful sentence recognition method of the present application. DETAILED DESCRIPTION

[0036] In order to facilitate the solution of the problems described in the background art, the embodiments of the present application provide a harmful sentence recognition method for private chat function of a dating software. The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0037] Please refer to Figure 1 The embodiments provide a harmful sentence recognition method for private chat function of a dating software, which comprises the following steps:

[0038] Step A: Establish a high-risk emoji list for storing part of emojis that are easily associated with bad content (whether in the case of emoji combination or in the case of combination with text content) in a specific context.

[0039] When an emoji is designed, the designer will give it an initial meaning. However, in the actual use process, these emojis will gradually develop new meanings. The number of these meanings is usually limited, and there is a difference in frequency of use. Therefore, in addition to the initial meaning, a frequency probability value is marked for each extended meaning of the same emoji. This frequency probability value is obtained by analyzing the sentences (for example, comments posted on social media) on the network that use the corresponding emoji.

[0040] Step B: Emojis are not processed in the form of images, but in the form of a special coded form, such as Unicode encoding, rich text format encoding, or custom encoding. Therefore, a mapping relationship between emojis and their codes is established, and the emojis existing in the message in the input box are replaced with the corresponding codes, so that the entire message in the input box is transformed into a whole string for processing;

[0041] Step C: Set detection conditions. If the input message does not meet the conditions, the input message is supplemented and combined. If the input message meets the conditions, the input message is directly detected.

[0042] Step D: Use the sliding window mechanism to detect the transformed input message. If the input message contains illegal words after inspection, the user is reminded to modify the input message.

[0043] Specifically, it further comprises:

[0044] Step E: Allow users to provide feedback to adjust the rules of semantic detection. If the user thinks the message is not risky but is considered harmful information, or thinks the information is harmful but is not processed by the identification system, feedback can be provided, and the auditor will update the high-risk emoji list based on this information.

[0045] At the same time, the frequency of use of each connotation of the emoji should be recalculated periodically, and the probability of use of the connotation should be updated to ensure that the latest usage trends of the connotation are covered.

[0046] Specifically, the emoji in step A has a unique corresponding code, and the code corresponding to the unique emoji corresponds to multiple connotations.

[0047] Specifically, step C includes:

[0048] Step C1, set the detection condition: no less than four bytes (generally, an emoji occupies 4 bytes in UTF-8 encoding; an English letter occupies 1 character, and a Chinese character occupies 3 characters; 4 characters are the maximum occupancy of a single input)

[0049] Step C2, if the input message does not meet the detection condition, the sent message and the input message will be combined until the length of the input message that does not pass the detection meets the detection condition, and it is considered as a message for judgment, in order to prevent the behavior of evading harmful information identification mechanism by breaking up a complete sentence. The number of previous messages taken is determined by the length of the previous message. If the length of the previous message is also too short, take one more, until the length of a single message meets the requirements, or the message is taken, or the maximum number of messages set in the program is reached.

[0050] Step C3, the message that meets the detection condition is directly detected.

[0051] Specifically, step D includes:

[0052] Step D1, set the sliding window to 6 code units (code unit refers to the smallest unit of a string in a message, such as a Chinese character, an emoji, two bytes, etc., which can be set according to the user's needs), and set the step size to one code unit, which can be set according to the user's needs.

[0053] Step D2, when the input message exceeds the capacity of the sliding window, the sliding window starts to slide the input message for detection. Starting from the beginning of the input message, the window size of the string is cut off each time, and the step size is one.

[0054] When the number of characters input by the user exceeds the size of the sliding window, the content in the sliding window is detected and sliding is started, starting from the beginning of the input content, and the string of the size of the window is intercepted each time the sliding is performed by one step. This step is repeatedly executed until all parts of the entire input content are covered by the sliding window and detected. In this way, the user can start detection while inputting content, and detection can be realized while inputting.

[0055] In step D3, a substring is generated in the string each time the sliding window slides, and if the substring contains an emoticon, the corresponding meaning group is found according to the emoticon code, the emoticon content is replaced with the meaning, and semantic detection is performed on the replaced message.

[0056] In step D4, after the semantic detection is completed, only harmless content can be sent out, and if harmful information is detected in the statement, the message sending is prevented, and the client is indicated the rule-breaking words in the input message.

[0057] Specifically, it further includes:

[0058] In step D5, steps D1-D4 are repeatedly executed, and if no harmful information is detected, the message is allowed to be sent out.

[0059] Specifically, the step D sliding window detection is performed simultaneously with the input message of the user, and the client is reminded in real time to modify the rule-breaking words detected in the input message. The client can know which words in the input message are rule-breaking words, and timely modification is made for the rule-breaking words.

[0060] Specifically, the step D3 further includes:

[0061] When the meaning group is replaced, the initial meaning of the emoticon is preferentially replaced into the sentence, and then the extended meaning is used for replacement and semantic detection is performed on the replaced message.

[0062] Specifically, the replacement order of the extended meaning is determined according to the use frequency probability value of the label, and the extended meaning with a high probability value is preferentially replaced.

[0063] In use:

[0064] Before detecting the input message, a high-risk emoticon list is established, which is used to store part of the emoticons that are easily associated with bad content in a specific context. A use frequency probability value is marked for each extended meaning of the same emoticon, and the use frequency probability value is obtained by analyzing the statements (for example, comments published in social media) on the network that use the corresponding emoticon.

[0065] The user replaces the emoticons in the message with unique corresponding codes while inputting information using the social software, thereby deforming the entire input information into a whole string for processing.

[0066] After the processing of the message is completed, the length of the message in the input chat box is judged to see if it reaches the detection condition. If the input message meets the condition of not less than four bytes, the next step of checking the processed message is directly performed. If the message does not meet the condition of not less than four bytes, the sent message and the input message are combined until the length of the input message that does not pass the detection meets the detection condition, and the message is considered as one message for judgment, so as to prevent the behavior of evading the harmful information identification mechanism by breaking a complete sentence. The number of previous messages is determined according to the length of the previous message. If the length of the previous message is also too short, one more previous message is taken until the length of a single message meets the requirement, or the messages are taken completely, or the maximum number of messages in the program is reached. The information that does not meet the condition is processed so that the message is convenient for detection.

[0067] The message that meets the detection condition is detected using a sliding window mechanism. The sliding window is set to six code units, and the step size of the sliding window is set to one code unit. When the number of characters input by the user exceeds the capacity size of the sliding window, the content in the sliding window is detected and starts to slide. Starting from the beginning of the input content, the string of the window size is intercepted each time the window slides, and the window slides one step each time. This step is repeated until all parts of the entire input content are covered and detected by the sliding window. This way can start detection when the user inputs the content, which can reduce the delay of the sentence being sent out compared to the usual pressing of the send key and then starting detection, and is suitable for the real-time output scene of chat.

[0068] A new substring is generated in the string each time the sliding window slides. Each substring is analyzed. If there is an emoticon in the substring, the corresponding connotative meaning of the emoticon is found according to its code, the emoticon content is replaced with the connotative meaning, and a judgment is made. When replacing, the initial meaning is preferentially selected to be replaced into the sentence. Then the connotative meaning is used for replacement and semantic detection. The replacement order of the connotative meaning is determined according to the previously marked frequency probability value of the connotative meaning. The connotative meaning with a high probability value is preferentially replaced. The message is detected semantically with respect to the replaced connotative meaning, so that any one of the connotative meanings of the used emoticon in the message does not produce meanings such as harassment, fraud, and violence.

[0069] After the detection of the message, only harmless messages can be sent out, if the statement is detected to have harmful information, the message sending is prevented, and the customer is pointed out the violation words in the input message. In the scheme, the input can be detected at the same time, so that the user can know the specific place in the message intercepted by the monitoring system in time, and correct the content input into the chat box according to the violation words.

[0070] In the use process, the method allows the user to feedback to adjust and optimize the judgment rules of semantic detection. If the user thinks that the message is risk-free but is considered to have harmful information, or thinks that the information is harmful but is not processed by the identification system, feedback can be given on it, and the audit personnel will update the high-risk list according to the information. At the same time, the use frequency of each connotation of the emoticon should also be regularly re-captured data to calculate and update the connotation use probability, to ensure that the latest use trend of the connotation is covered.

[0071] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and modifications can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A harmful sentence recognition method for a snap software private chat function, characterized by, It comprises the following steps: Step A: Establishing an emoji list, marking the use frequency probability value for each connotative meaning of the same emoji, and the emoji has a unique corresponding code, and the code corresponding to the unique emoji corresponds to multiple connotative meanings; Step B: Replace the emoji in the input message with the corresponding code; Step C: Set the detection condition to be no less than four bytes; If the input message does not meet the detection condition, the sent message and the input message will be combined until the length of the input message that does not pass the detection meets the detection condition; the input message that meets the condition is detected; Step D: Set the sliding window to be 6 code units, and set the step size to be one code unit; when the input message exceeds the capacity size of the sliding window, the sliding window starts to slide and detect the input message, starting from the beginning of the input message, each time sliding a string of window size, and sliding one step each time; each time the sliding window slides, a substring is generated in the string, if the substring contains an emoji, find the corresponding meaning group according to the emoji code, replace the emoji content with the meaning, and then perform semantic detection on the replaced message; when replacing the meaning group, the initial meaning of the emoji is preferentially selected and replaced into the sentence, and then the connotative meaning is used for replacement and semantic detection on the replaced message; if the sentence is detected to contain harmful information, the message sending is prevented, and the customer is pointed out the illegal words in the input message, and the step is repeated, if no harmful information is detected, the message is allowed to be sent out.

2. The method for identifying harmful sentences for private chat function of a software according to claim 1, wherein, It also comprises: Step E: Update the high-risk emoji list according to the customer's use feedback, and periodically update the use frequency of each connotative meaning of the emoji. 3.The method for identifying harmful sentences for private chat function of screen capture software according to claim 1, wherein, The step D sliding window detection is performed simultaneously with the user input message, and the customer is reminded in real time. 4.The method for identifying harmful sentences in a private chat function of a screen capture software according to claim 1, wherein, The replacement order of the connotative meaning is determined according to the marked use frequency probability value, and the connotative meaning with high probability value is preferentially replaced.

Images