Method, device, medium and electronic equipment for identifying specific type of word segmentation

Abstract

The application provides a specific type word segmentation identification method, a specific type word segmentation identification device, a computer readable storage medium and an electronic device; relates to the technical field of computers; comprising: obtaining a word segmentation set of a text to be analyzed; determining the reference score of a target word segmentation in the word segmentation set based on each word segmentation in the word segmentation set; obtaining relevant word segmentation corresponding to adjacent word segmentation of the target word segmentation from a preset knowledge graph, and replacing the target word segmentation in the word segmentation set with the relevant word segmentation; determining the reference score of the relevant word segmentation based on each word segmentation in the word segmentation set; and performing specific type identification on the target word segmentation according to the reference score of the target word segmentation and the reference score of the relevant word segmentation. As can be seen, by implementing the embodiments of the application, multi-dimensional specific type evaluation is obtained through multi-dimensional data calculation of word segmentation, specific type identification is performed in combination with multi-dimensional evaluation, the limitations of a word table are avoided, multi-dimensional analysis is realized based on artificial intelligence, and the identification accuracy of specific type word segmentation is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular, to a specific type word segmentation identification method, a specific type word segmentation identification device, a computer readable storage medium and an electronic device. BACKGROUND

[0002] With the development of computer technology, users can chat through instant messaging software. In the process of chatting, when the user sends some sensitive words in the phrase blacklist, it will be generally processed by the software, for example, the software will replace the sensitive words in the text with some alternative characters (such as ***). However, in order to chat normally with the other party, the user will also use some specific type words (such as Hong Kong dollars) to replace the sensitive words (such as pen).

[0003] The specific type word recognition method for sensitive words usually includes: constructing a specific type word table of sensitive words, when the user input text contains a phrase that hits the word table, the phrase can be identified as a specific type word of sensitive words. However, this method has the limitation of word quantity in the word table. When the user agrees on a new specific type word to replace some sensitive words, and the new specific type word is not included in the above-mentioned word table, it cannot be identified. It can be seen that the above-mentioned method has the problem of low specific type word recognition accuracy.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The purpose of the present application is to provide a specific type word segmentation identification method, a specific type word segmentation identification device, a computer readable storage medium and an electronic device, which can obtain specific type evaluation of word segmentation in multiple dimensions through multi-dimensional analysis of word segmentation in text, so as to identify specific type word segmentation by combining the evaluation of multiple dimensions, avoiding the limitation of the word table in the prior art. And, based on artificial intelligence technology, multi-dimensional analysis of word segmentation in text can be realized, and the recognition accuracy of specific type word segmentation can be improved.

[0006] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0007] According to an aspect of the present application, a specific type word segmentation identification method is provided, comprising:

[0008] Obtaining a word segmentation set of the text to be analyzed;

[0009] The reference score of the target word in the word set is determined based on each word in the word set, and the reference score of the target word is used for the specific type evaluation of the target word;

[0010] The related word corresponding to the adjacent word of the target word is obtained from the preset knowledge graph, and the target word in the word set is replaced by the related word;

[0011] The reference score of the related word is determined based on each word in the word set, and the reference score of the related word is used for the specific type evaluation of the target word;

[0012] The target word is identified according to the reference score of the target word and the reference score of the related word.

[0013] In an exemplary embodiment of the present application, the related word is a random word in a word group or a word with the highest calling frequency in the word group, wherein the word group is a word group to which the adjacent word belongs in the preset knowledge graph.

[0014] According to an aspect of the present application, a specific type word identification device is provided, comprising a word obtaining unit, a parameter determining unit, a word replacing unit and a specific type word identification unit, wherein:

[0015] The word obtaining unit is configured to obtain a word set of a text to be analyzed;

[0016] The parameter determining unit is configured to determine a reference score of a target word in the word set based on each word in the word set, and the reference score of the target word is used for the specific type evaluation of the target word;

[0017] The word replacing unit is configured to obtain a related word corresponding to an adjacent word of the target word from a preset knowledge graph, and replace the target word in the word set with the related word;

[0018] The parameter determining unit is further configured to determine a reference score of the related word based on each word in the word set, and the reference score of the related word is used for the specific type evaluation of the target word;

[0019] The specific type word identification unit is configured to identify the target word according to the reference score of the target word and the reference score of the related word.

[0020] In an exemplary embodiment of the present application, the word obtaining unit obtains a word set of a text to be analyzed, comprising:

[0021] Extracting a specific phrase with a frequency higher than a frequency threshold in the sample conversation;

[0022] Performing word segmentation processing on the obtained text to be analyzed to obtain a reference word set;

[0023] reorganize the reference segmentation set based on the specific phrases to obtain a segmentation set.

[0024] In an example embodiment of the present application, the device further comprises:

[0025] a session reading unit, configured to read at least one set of historical sessions before the segmentation obtaining unit extracts the specific phrases with a frequency higher than the frequency threshold from the sample session, and filter the sample session from the at least one set of historical sessions according to the number of texts; wherein the sample session contains a number of texts greater than a preset number of texts.

[0026] In an example embodiment of the present application, the segmentation obtaining unit extracts the specific phrases with a frequency higher than the frequency threshold from the sample session, comprising:

[0027] performing word-by-word segmentation on the sample texts in the sample session to obtain a set of words used to constitute the sample texts;

[0028] combining adjacent words in the set of words according to the set adjacent number and the word position relationship in the sample texts to obtain a set of phrases corresponding to the sample texts;

[0029] obtaining all the sets of phrases in the sample session to obtain a total set of phrases;

[0030] calculating the frequency of each phrase in the total set of phrases according to the total number of phrases in the total set of phrases;

[0031] determining the phrases with a frequency higher than the frequency threshold as the specific phrases.

[0032] In an example embodiment of the present application, the segmentation obtaining unit performs segmentation processing on the obtained text to be analyzed to obtain a reference segmentation set, comprising:

[0033] selecting the text to be analyzed according to the text publishing order of the current session;

[0034] performing segmentation processing on the text to be analyzed to obtain the reference segmentation set.

[0035] In an example embodiment of the present application, the segmentation obtaining unit reorganizes the reference segmentation set based on the specific phrases to obtain a segmentation set, comprising:

[0036] merging the to-be-merged segments in the reference segmentation set based on the specific phrases to realize segmentation reorganization and obtain the segmentation set;

[0037] wherein the to-be-merged segments are segments in the reference segmentation set that hit the same specific phrase, and the number of segments in the segmentation set is less than or equal to the number of segments in the reference segmentation set.

[0038] In an example embodiment of the present application, the word replacement unit obtains the relevant word corresponding to the adjacent word of the target word from the preset knowledge graph, including:

[0039] Comparing the preset knowledge graph with the adjacent word of the target word, an alignment result is obtained.

[0040] If the alignment result indicates that the adjacent word hits the preset knowledge graph, the relevant word corresponding to the adjacent word of the target word is obtained from the preset knowledge graph.

[0041] In an example embodiment of the present application, the word replacement unit obtains the relevant word corresponding to the adjacent word of the target word from the preset knowledge graph, including:

[0042] Determining the word group in which the adjacent word is located from the preset knowledge graph.

[0043] Selecting the word with the same part of speech as the adjacent word from the word group as the relevant word; wherein the relevant word is a random word in the word group or the word with the highest calling frequency in the word group.

[0044] In an example embodiment of the present application, if the number of relevant words is greater than 1, the parameter determination unit determines the reference score of the relevant word based on each word in the word set, including:

[0045] Determining the reference score of each relevant word after being replaced in the word set, and obtaining a reference score set;

[0046] Calculating the weighted average of the reference score set as the reference score of the relevant word.

[0047] In an example embodiment of the present application, the specific type word identification unit identifies the target word according to the reference score of the target word and the reference score of the relevant word, including:

[0048] Determining the difference between the reference score of the target word and the reference score of the relevant word;

[0049] Identifying the target word according to the difference and the reference score of the target word.

[0050] In an example embodiment of the present application, the specific type word identification unit identifies the target word according to the difference and the reference score of the target word, including:

[0051] Calculating the weighted sum of the difference and the reference score of the target word according to a preset weight value;

[0052] If the weighted sum is greater than or equal to the preset threshold, it is determined that the target word belongs to a specific type; the specific type is used to specifically indicate that the target word is a rewritten word of a preset sensitive word.

[0053] In an example embodiment of the present application, the parameter determination unit determines the reference score of the target word in the word set based on each word in the word set, including:

[0054] predicting, based on the prediction model, the appearance probability of the target word according to the previous word adjacent to the target word;

[0055] predicting, based on the classification model, the specific probability that the target word belongs to a specific type;

[0056] determining the reference score of the target word according to the appearance probability and the specific probability.

[0057] In an example embodiment of the present application, the above device further includes:

[0058] The model training unit is configured to obtain at least one sample preset word in a preset knowledge graph; each of the at least one sample preset word corresponds to a word adjacent to the target word; input the at least one sample preset word into the classification model to train the classification model to distinguish the at least one sample preset word as a specific type, and obtain a discrimination result; and update the classification model according to the discrimination result.

[0059] According to an aspect of the present application, an electronic device is provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above method via the executable instructions.

[0060] According to an aspect of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the above method.

[0061] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the method provided in the various optional implementation manners.

[0062] The example embodiments of the present application can have the following partial or all beneficial effects:

[0063] In the method for identifying a specific type of word segmentation provided by an example embodiment of the present application, a word segmentation set of a text to be analyzed can be obtained; a reference score of a target word segmentation in the word segmentation set is determined based on each word segmentation in the word segmentation set, and the reference score of the target word segmentation is used for specific type evaluation of the target word segmentation; a related word segmentation corresponding to a neighboring word segmentation of the target word segmentation is obtained from a preset knowledge graph, and the target word segmentation in the word segmentation set is replaced by the related word segmentation; a reference score of the related word segmentation is determined based on each word segmentation in the word segmentation set, and the reference score of the related word segmentation is used for specific type evaluation of the target word segmentation; and the target word segmentation is identified according to the reference score of the target word segmentation and the reference score of the related word segmentation. According to the above description, the present application can obtain specific type evaluation of a word segmentation in multiple dimensions through multi-dimensional analysis of the word segmentation in the text, so that the word segmentation can be identified in specific types in combination with the evaluation in multiple dimensions, thereby avoiding the limitation of the word table in the prior art. Furthermore, the multi-dimensional analysis of the word segmentation in the text can be realized based on artificial intelligence technology, thereby improving the identification accuracy of the specific type of word segmentation. In another aspect of the present application, the target word segmentation can be replaced based on a preset word segmentation in the preset knowledge graph, so that the reference scores before and after the replacement are used as multi-dimensional evaluation of the target word segmentation, thereby realizing mining of the specific type of word segmentation. Since the present application does not need to construct a word table of the specific type of word segmentation, the word table of the specific type of word segmentation can be avoided to occupy computer resources, thereby facilitating improvement of the utilization rate of computer resources.

[0064] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0065] The drawings incorporated in the specification and constituting a part of the specification illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0066] Figure 1 A schematic diagram of an exemplary system architecture of a method for identifying a specific type of word segmentation and a device for identifying a specific type of word segmentation to which the embodiments of the present application can be applied is shown;

[0067] Figure 2 A structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application is shown;

[0068] Figure 3 A flowchart of a method for identifying a specific type of word segmentation according to an embodiment of the present application is schematically shown;

[0069] Figure 4 A schematic diagram of a conversation interface is shown according to an embodiment of the present application;

[0070] Figure 5 A schematic diagram of a classification model structure before reorganization of word segmentation is shown according to an embodiment of the present application;

[0071] Figure 6 A schematic diagram of a prediction model structure is shown according to an embodiment of the present application;

[0072] Figure 7 A schematic diagram of a classification model structure is shown according to an embodiment of the present application;

[0073] Figure 8 A schematic diagram of a classification model structure based on model training is shown according to an embodiment of the present application;

[0074] Figure 9 A flowchart of a recognition method for a specific type of word segmentation is shown according to an embodiment of the present application;

[0075] Figure 10 A block diagram of a recognition device for a specific type of word segmentation is shown according to an embodiment of the present application. DETAILED DESCRIPTION

[0076] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations can be implemented in any

[0077] In addition, the accompanying drawings are included to provide a thorough understanding of embodiments of the application and are not intended to be in any way limiting of the application. Unless otherwise noted, identical or similar components are identified with the same reference numerals throughout the text. The following detailed description makes reference to the accompanying drawings.

[0078] Figure 1 A schematic diagram of a system architecture of an exemplary application environment of a method for identifying a specific type of word segmentation and an apparatus for identifying a specific type of word segmentation to which embodiments of the present application can be applied is shown.

[0079] As shown in Figure 1 the system architecture 100 can include one or more of terminal devices 101, 102, 103, a network 104, and a server cluster 105. The network 104 serves as a medium to provide communication links between the terminal devices 101, 102, 103 and the server cluster 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, among others. The terminal devices 101, 102, 103 can be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, and tablet computers, among others. It should be understood that Figure 1 the number of terminal devices, networks, and servers in

[0080] The method for identifying a specific type of word segmentation provided by embodiments of the present application can be performed by any of the terminal devices 101, 102, 103 or the servers of the server cluster 105. Accordingly, the apparatus for identifying a specific type of word segmentation is generally provided in the servers of the server cluster 105 or the terminal devices 101, 102, 103. For example, in an exemplary embodiment, any of the servers of the server cluster 105 can obtain a set of word segments of a text to be analyzed; determine a reference score of a target word segment in the set of word segments based on each word segment in the set of word segments, the reference score of the target word segment being used for a specific type of evaluation of the target word segment; obtain a related word segment corresponding to a neighboring word segment of the target word segment from a pre-set knowledge graph, and replace the target word segment in the set of word segments with the related word segment; determine a reference score of the related word segment based on each word segment in the set of word segments, the reference score of the related word segment being used for the specific type of evaluation of the target word segment; and identify the target word segment according to the reference score of the target word segment and the reference score of the related word segment.

[0081] The server can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and basic cloud computing services such as big data and artificial intelligence platform. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0082] The present application can apply cloud computing in cloud technology to perform specific type segmentation recognition on the text to be analyzed. The cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, network, etc. in a wide area network or a local area network to realize data calculation, storage, processing and sharing. The cloud technology is a general term of network technology, information technology, integration technology, management platform technology, application technology, etc. based on cloud computing business model application, which can form a resource pool and be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources, such as video websites, picture websites and more portals. With the high development and application of the Internet industry, every item in the future may have its own identification mark and needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data need strong system support, which can only be realized through cloud computing.

[0083] Cloud computing in cloud technology is a computing mode that distributes computing tasks on a large number of computing resources, so that various application systems can obtain computing power, storage space and information services according to needs. The network providing resources is called "cloud". The resources in the "cloud" are infinitely expandable to users and can be obtained at any time, used on demand, expanded at any time and paid according to use. As a basic capability provider of cloud computing, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) is established, and a plurality of types of virtual resources are deployed in the resource pool for external customers to select and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, network devices. According to logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer is deployed above the PaaS layer, or the SaaS can be directly deployed on the IaaS. PaaS is a platform for software running, such as databases, web containers, etc. SaaS is various business software, such as web portal websites, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0084] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware level technology and software level technology. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. several major directions.

[0085] Cloud computing can perform corresponding calculations based on artificial intelligence. Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0086] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, and so on. It is a specialized research on how a computer simulates or implements human learning behaviors to acquire new knowledge or skills, reorganize existing knowledge structure, and constantly improve its own performance. Machine learning is the core of artificial intelligence and a fundamental approach to making computers intelligent. It is applied to various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and inductive learning.

[0087] Figure 2 A structural diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown.

[0088] It should be noted that, Figure 2 The computer system 200 of the electronic device shown is only an example and should not impose any limitation on the functions and use range of embodiments of the present application.

[0089] As Figure 2 shown, the computer system 200 includes a central processing unit (CPU) 201 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 202 or programs loaded from a storage portion 208 into a random access memory (RAM) 203. Various programs and data required for system operation are also stored in the RAM 203. The CPU 201, the ROM 202, and the RAM 203 are connected to each other through a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0090] The following components are connected to the I / O interface 205: an input portion 206 including a keyboard, a mouse, and the like; an output portion 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 208 including a hard disk, and the like; and a communication portion 209 including a network interface card such as a LAN card, a modem, and the like. The communication portion 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 210 as needed, so that a computer program read therefrom is installed in the storage portion 208 as needed.

[0091] In particular, according to embodiments of the present application, the processes described below with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 209, and / or installed from the removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, various functions defined in the methods and apparatuses of the present application are performed.

[0092] In the prior art, the identification method of the specific type of words of sensitive words usually can include the following ways:

[0093] I. Construct a word list of the specific type of words of sensitive words. When there is a phrase in the text input by the user that hits the word list, the phrase can be identified as the specific type of words of sensitive words. However, this approach has the limitation of the amount of words in the word list. When the user agrees to replace some sensitive words with a new specific type of words, and the new specific type of words is not included in the above-mentioned word list, it cannot be identified.

[0094] II. Perform semantic recognition on the text input by the user. If the recognition result indicates that there is a word in the word list that has a high similarity with a phrase in the text, the phrase can be identified as the specific type of words. However, this approach also needs to constantly supplement the word list to maintain the recognition accuracy, which has the problem of high maintenance cost of the word list. Moreover, semantic recognition can also have recognition errors (e.g., identifying "drug abuse" and "smoking" as similar words), and over-reliance on semantic recognition can also affect the recognition accuracy of the specific type of words.

[0095] III. Add the pinyin of each word in the word list to enrich the word list, and then identify the specific type of words based on the word list. However, there are many homophones of words. If a phrase is identified as a specific type of word based on the pinyin, it is easy to cause recognition errors.

[0096] Based on the above problems, the present example embodiment provides an identification method for specific type of word segmentation. Please refer to Figure 3 , Figure 3 The flowchart of the identification method for specific type of word segmentation according to one embodiment of the present application is schematically shown. As Figure 3 shown, the identification method for specific type of word segmentation can include steps S310-S350.

[0097] Step S310: Obtain a word segmentation set of the text to be analyzed.

[0098] Step S320: determining a reference score of the target word in the word set based on each word in the word set, the reference score of the target word being used for the specific type evaluation of the target word.

[0099] Step S330: obtaining a related word corresponding to the adjacent word of the target word from the preset knowledge graph, and replacing the target word in the word set with the related word.

[0100] Step S340: determining a reference score of the related word based on each word in the word set, the reference score of the related word being used for the specific type evaluation of the target word.

[0101] Step S350: performing specific type identification on the target word according to the reference score of the target word and the reference score of the related word.

[0102] Implementation Figure 3 The method shown can obtain specific type evaluation of the word in multiple dimensions through multi-dimensional analysis of the word in the text, so that the word can be identified in specific types by combining the evaluation of multiple dimensions, thereby avoiding the limitations of the word table in the prior art. In addition, multi-dimensional analysis of the word in the text can be realized based on artificial intelligence technology, which improves the identification accuracy of the specific type word. In addition, the target word can be replaced based on the preset word in the preset knowledge graph (such as a knowledge graph), so that the reference scores before and after the replacement are used as multi-dimensional evaluation of the target word, thereby realizing mining of the specific type word. Since the present application does not need to construct a word table of a specific type word, the word table of the specific type word can be avoided to occupy computer resources, thereby facilitating the utilization of computer resources.

[0103] Next, the above steps of the present example embodiment will be described in more detail.

[0104] In step S310, a word set of a text to be analyzed is obtained.

[0105] Specifically, the text to be analyzed can be a text input by a user, and the present application can be applied to a conversation window of an instant messaging software. For a conversation, the text input by a conversation party in an input box can be the text to be analyzed. The text input by the conversation party can include characters, characters, numbers, letters, etc., and the present embodiment is not limited. The word set of the text to be analyzed refers to the word set of the characters in the text.

[0106] In addition, the number of the segmented words in the segmented word set of the text to be analyzed is at least one, and embodiments of the present application are not limited, for example, if the text to be analyzed is "who can change the exchange rate of the pen", the corresponding segmented word set can be [who, can, change, pen, exchange rate]. Wherein, "pen" can be a rewritten word of the sensitive word "Hong Kong dollars". Wherein, the rewritten word of the sensitive word is the rewritten replacement of the sensitive word by the user.

[0107] As an optional embodiment, the segmented word set of the text to be analyzed is obtained, including: extracting a specific phrase with a frequency higher than a frequency threshold in a sample session; performing segmented word processing on the obtained text to be analyzed to obtain a reference segmented word set; and performing segmented word reorganization on the reference segmented word set based on the specific phrase to obtain the segmented word set.

[0108] Specifically, the frequency threshold is used to limit the minimum segmented word frequency.

[0109] It can be seen that by implementing this optional embodiment, the specific phrase can be extracted based on the real user session, and the segmented word set can be corrected according to the specific phrase, which can improve the accuracy of segmented words and thus improve the recognition accuracy of specific types of segmented words.

[0110] As an optional embodiment, before extracting the specific phrase with a frequency higher than the frequency threshold in the sample session, the above method further includes: reading at least one set of historical sessions, and selecting a sample session from the at least one set of historical sessions according to the number of texts; wherein the number of texts contained in the sample session is greater than a preset number of texts.

[0111] Specifically, the historical session is a stored real user session in a unit of time (such as 1 year). In addition, the number of texts is the number of dialogue boxes included in the session, and the number of dialogue boxes includes the number of dialogue boxes sent by one party of the dialogue and the number of dialogue boxes sent by the other party of the dialogue. According to the number of texts, the sample session is selected from the at least one set of historical sessions, including: selecting a historical session with a number of texts greater than a preset number of texts (such as 10) from the at least one set of historical sessions as a sample session.

[0112] It can be seen that by implementing this optional embodiment, the historical sessions can be screened to obtain effective sample sessions, which can be beneficial to improve the recognition accuracy of the specific phrase in the real user session and can reduce the waste of computing resources by ineffective sample sessions.

[0113] As an optional embodiment, extracting the specific phrase with the frequency higher than the frequency threshold in the sample session comprises: performing word-by-word segmentation on the sample text in the sample session to obtain a word set used to constitute the sample text; combining adjacent words in the word set according to the set adjacent number and the word position relationship in the sample text to obtain a phrase set corresponding to the sample text; obtaining all phrase sets in the sample session to obtain a total phrase set; calculating the occurrence frequency of each phrase in the total phrase set according to the total phrase quantity of the total phrase set; and determining the phrase with the occurrence frequency higher than the frequency threshold (for example, 10 times) as the specific phrase.

[0114] Specifically, the number of words in the word set is consistent with the number of words in the sample text. For example, performing word-by-word segmentation on the sample text "who can change the exchange rate of a pen" can obtain the word set [who, can, change, steel, pen, exchange rate]. The adjacent number can be n and the set adjacent number can be one or more, where n is a positive integer. For example, the set adjacent number can include 2 and 4, that is, combining every two adjacent words in the word set and combining every four adjacent words in the word set to obtain the phrase set containing 2-word phrases (for example, "who can", "can change", "change steel", "steel pen", "pen exchange", "exchange rate") and 4-word phrases (for example, "who can change", "can change steel", "change steel pen", "steel pen exchange", "pen exchange rate").

[0115] In addition, the specific phrase used to correct the word segmentation set can be defined as a user frequently used phrase.

[0116] It can be seen that by implementing the optional embodiment, the real user session can be used as a sample session for specific phrase word segmentation, so that the specific phrase can be used as an optimization basis for the word segmentation set to improve the word segmentation accuracy and reduce the recognition error of a specific type of word segmentation due to word segmentation error.

[0117] As an optional embodiment, performing word segmentation processing on the obtained text to be analyzed to obtain a reference word segmentation set comprises: selecting the text to be analyzed according to the text publishing order of the current session; and performing word segmentation processing on the text to be analyzed to obtain the reference word segmentation set.

[0118] Specifically, the current session can be a session to which the text to be analyzed belongs, and the text publishing order of the current session can be obtained based on the sending time of each text in the session; wherein, the session refers to a paragraph composed of a series of dialogues, which essentially belongs to a dialogue. For example, if the current session includes: the text "Hello, I submitted a patent a few days ago, how to query the status" with a sending time of January 1, 2020, 00:00, the text "Log in to the website" with a sending time of January 1, 2020, 00:10, and the text "OK" with a sending time of January 1, 2020, 00:20. It can be seen that the text publishing order can be "Hello, I submitted a patent a few days ago, how to query the status" -> "Log in to the website" -> "OK".

[0119] Based on this, the text to be analyzed is selected according to the text publishing order of the current session, which includes: determining the text that has not been recognized in the current session according to the text publishing order of the current session, and determining the text as the text to be analyzed.

[0120] Please refer to Figure 4 , Figure 4 The schematic diagram of the session interface according to an embodiment of the present application is schematically shown. As Figure 4 shown, the session interface can include at least one group of sessions, such as session 410 and session 420. Session 410 and session 420 are used to discuss different topics respectively. Among them, session 410 includes the following texts: "Hello, I am applying for a patent and want to consult a problem", "What problem", "How to write a disclosure report", "You can download a template on the website", "OK". Session 420 includes the following texts: "Hello, I submitted a patent a few days ago, how to query the status", "Log in to the website", "OK". Among them, for the session, not only can it be divided based on the topic, but also can be divided based on the time span, for example, session 410 and session 420 not only correspond to different topics, but also can exist a time span (such as 1 month), and the session can be divided by detecting the time span between the texts.

[0121] In addition, the text to be analyzed is subjected to word segmentation processing to obtain a reference segmentation set, which includes: performing word segmentation processing on the text to be analyzed according to a maximum matching segmentation algorithm / shortest path segmentation algorithm / generative model segmentation algorithm / discriminative model segmentation algorithm to obtain a reference segmentation set, which is not limited in the embodiments of the present application.

[0122] It can be seen that by implementing this optional embodiment, the corresponding segmentation set can be obtained through the word segmentation processing of the text to be analyzed, so that the segmentation in it can be analyzed in turn to identify the specific type of segmentation.

[0123] In step S320, a reference score of a target word in the word set is determined based on each word in the word set, and the reference score of the target word is used for the specific type evaluation of the target word.

[0124] Specifically, the first dimension can be used for forward description of the target word, and the second dimension can be used for lateral description of the target word. The reference score of the target word is used to represent the possibility of the target word appearing in the text to be analyzed, and the reference score of the related word is used to represent the possibility of the related word appearing in the text to be analyzed. The smaller the reference score of the target word / the reference score of the related word, the smaller the possibility of the target word / related word appearing in the text to be analyzed. The target word is any word in the word set. According to steps S310-S350, each word in the word set of the text to be analyzed can be sequentially taken as a target word for specific type recognition, so as to improve the detection rate of specific type words, and facilitate specific processing of specific type words, thereby maintaining a healthy conversation environment.

[0125] As an optional embodiment, the reference word set is reorganized based on the specific phrase to obtain the word set, including: merging the to-be-merged words in the reference word set based on the specific phrase to realize reorganization of the words and obtain the word set; wherein the to-be-merged words are words in the reference word set that hit the same specific phrase, and the number of words in the word set is less than or equal to the number of words in the reference word set.

[0126] Specifically, the to-be-merged words in the reference word set are merged based on the specific phrase, including: merging the words in the reference word set that hit the specific phrase as to-be-merged words based on the specific phrase, and the new word obtained after merging the to-be-merged words is consistent with the specific phrase.

[0127] Please refer to Figure 5 , Figure 5 The classification model structure schematic diagram before reorganization of words according to an embodiment of the present application is schematically shown. As shown in Figure 5 , the classification model can include: an input layer 510, a first feature extraction layer 520, a full connection layer 530, a second feature extraction layer 540, a full connection layer 550, a third feature extraction layer 560, and an output layer 570.

[0128] Specifically, each word segment in the word segment set [who, can, exchange, pen, exchange rate, discount] can be input into the first feature extraction layer 520 through the input layer 510 in sequence. The first feature extraction layer 520 extracts the feature vectors of [who, can, exchange, pen, exchange rate, discount] and inputs them into the fully connected layer 530. The fully connected layer 530 can associate the received feature vectors, so that the associated feature vectors can fuse some features of other feature vectors and input the association result into the second feature extraction layer 540 for further feature extraction. Furthermore, the second feature extraction layer 540 can input the feature extraction into the fully connected layer 550, so that the fully connected layer 550 further associates the received feature vectors and inputs the association result into the third feature extraction layer 560, so that the third feature extraction layer 560 performs further feature extraction and inputs it into the output layer 570, so that the output layer 570 can predict the specific probability that the target word segment belongs to a specific type. According to the specific probability, it can be determined whether it belongs to a specific type. For example, the specific probabilities corresponding to [who, can, exchange, pen, exchange rate, discount] are [0%, 0%, 100%, 100%, 0%, 0%] respectively. Therefore, it can be determined that the determination results corresponding to [who, can, exchange, pen, exchange rate, discount] can be [no, no, yes, yes, no, no] respectively.

[0129] However, there is a word segmentation error for "exchange, pen". Based on the above embodiments, the to-be-merged word segments "exchange" and "pen" in the reference word segment set can be merged based on a specific phrase to achieve word segmentation recombination and obtain the word segment set [who, can, exchange pen, exchange rate, discount]. Furthermore, based on the correct word segmentation result, the recognition accuracy of specific type word segments can be improved. After the word segmentation recombination, the specific probabilities of each word segment can be calculated based on Figure 7 the classification model structure diagram shown.

[0130] Among them, the first feature extraction layer 520 can be set as a neural network (Recurrent Neural Network, RNN), gated recurrent unit (GRU), or long short-term memory network (Long Short-Term Memory, LSTM) for processing sequence data.

[0131] It should be noted that the first feature extraction layer 520, the second feature extraction layer 540, and the third feature extraction layer 560 correspond to different feature extraction parameters. The fully connected layer 530 and the fully connected layer 550 also correspond to different network parameters.

[0132] It can be seen that implementing this optional embodiment can avoid word segmentation errors based on word segmentation recombination to improve the recognition accuracy of specific type word segments.

[0133] As an optional embodiment, a reference score for the target segment in the segmentation set is determined based on each segment in the segmentation set, including: predicting the occurrence probability of the target segment based on the preceding segment adjacent to the target segment using a prediction model; predicting a specific probability that the target segment belongs to a specific type based on a classification model; and determining the reference score of the target segment based on the occurrence probability and the specific probability.

[0134] Specifically, the prediction model predicts the probability of occurrence of the target word based on the preceding word adjacent to the target word. This includes: inputting the word set into the prediction model and shifting it one position to the right to facilitate the prediction model's prediction of the probability of occurrence of the next word based on the input word; and then, predicting the probability of occurrence of the target word based on the preceding word adjacent to the target word. The prediction model can be a language model, which is an abstract mathematical model of language based on objective linguistic facts.

[0135] Please see Figure 6 , Figure 6 A schematic diagram illustrating the structure of a prediction model according to an embodiment of this application is shown. Figure 6 As shown, the prediction model may include: an input layer 610, a first feature extraction layer 620, a fully connected layer 630, a second feature extraction layer 640, a fully connected layer 650, a third feature extraction layer 660, and an output layer 670.

[0136] Specifically, each segmentation word in the word segmentation set [who, can, exchange for a pen, exchange rate, discount] can be sequentially input into the first feature extraction layer 620 through the input layer 610. The first feature extraction layer 620 extracts the feature vectors of [who, can, exchange for a pen, exchange rate, discount] and inputs them into the fully connected layer 630. The fully connected layer 630 can associate the received feature vectors so that the associated feature vectors can incorporate some features from other feature vectors, and the association result is input into the second feature extraction layer 640 for further feature extraction. Then, the second feature extraction layer 640 can input the feature extraction into the fully connected layer 650 so that the fully connected layer 650 can further associate the received feature vectors and input the association result into the third feature extraction layer 660 so that the third feature extraction layer 660 can perform further feature extraction and input it into the output layer 670 so that the output layer 670 can predict the occurrence probability of the target segmentation word based on the adjacent preceding segmentation word. For example, predict the probability of "can" based on "who", predict the probability of "exchange pen" based on "can", predict the probability of "exchange pen" based on "exchange pen", and predict the probability of "discount" based on "exchange rate".

[0137] The first feature extraction layer 620 can be configured as a neural network (RNN), gated recurrent unit (GRU), or long short-term memory (LSTM) for processing sequence data.

[0138] It should be noted that the first feature extraction layer 620, the second feature extraction layer 640, and the third feature extraction layer 660 correspond to different feature extraction parameters. The fully connected layer 630 and the fully connected layer 650 also correspond to different network parameters.

[0139] In addition, the above method may also include: inputting sample text into the prediction model in an unsupervised manner to train the prediction model, thereby improving the accuracy of the prediction model in predicting the next word based on the previous word of the text.

[0140] Specifically, predicting the specific probability that a target word belongs to a specific type based on a classification model includes: extracting features from the target word to obtain the feature vector corresponding to the target word, and predicting the specific probability that the target word belongs to a specific type based on the feature vector.

[0141] Please see Figure 7 , Figure 7 A schematic diagram illustrating a classification model structure according to an embodiment of this application is shown. Figure 7 As shown, the classification model may include: an input layer 710, a first feature extraction layer 720, a fully connected layer 730, a second feature extraction layer 740, a fully connected layer 750, a third feature extraction layer 760, and an output layer 770. It should be noted that the parameters in the feature extraction layers of the prediction model are different from those in the feature extraction layers of the classification model, and the parameters in the fully connected layers of the prediction model are also different from those in the fully connected layers of the classification model.

[0142] Specifically, each word segmentation set [who, can, exchange for a pen, exchange rate, discount] can be sequentially input into the first feature extraction layer 720 through the input layer 710. The first feature extraction layer 720 extracts the feature vectors of [who, can, exchange for a pen, exchange rate, discount] and inputs them into the fully connected layer 730. The fully connected layer 730 can associate the received feature vectors so that the associated feature vectors can incorporate some features from other feature vectors, and the association result is input into the second feature extraction layer 740 for further feature extraction. Then, the second feature extraction layer 740 can input the feature extraction into the fully connected layer 750 so that the fully connected layer 750 can further associate the received feature vectors and input the association result into the third feature extraction layer 760 so that the third feature extraction layer 760 can perform further feature extraction and input it into the output layer 770. This allows the output layer 770 to predict the specific probability that the target word belongs to a specific type, and based on the specific probability, it can determine whether it belongs to a specific type. For example, the specific probabilities corresponding to [who, can, exchange for a pen, exchange rate, discount] are [0%, 0%, 100%, 0%, 0%], so the judgment results corresponding to [who, can, exchange for a pen, exchange rate, discount] can be [no, no, yes, no, no].

[0143] The first feature extraction layer 720 can be configured as a neural network (RNN), gated recurrent unit (GRU), or long short-term memory (LSTM) for processing sequence data.

[0144] It should be noted that the first feature extraction layer 720, the second feature extraction layer 740, and the third feature extraction layer 760 correspond to different feature extraction parameters. The fully connected layer 730 and the fully connected layer 750 also correspond to different network parameters.

[0145] In addition, the above method may also include: inputting the labeled sample text into a classification model in a supervised manner to train the classification model, thereby improving the classification model's accuracy in recognizing specific types of classification. In this case, each word segment in the labeled sample text corresponds to a labeling result, which is used to characterize whether the word segment is a specific type of word segmentation.

[0146] In addition, the reference score for the target word segmentation is determined based on the occurrence probability and the specific probability, including: calculating the weighted sum of the occurrence probability and the specific probability based on the preset weights of the occurrence probability and the specific probability, and determining the weighted sum as the reference score for the target word segmentation; wherein, the preset weights can be represented by constants.

[0147] As can be seen, by implementing this optional embodiment, the target word segmentation can be probabilistically predicted through prediction and classification models. The reference score of the target word segmentation determined based on the above occurrence probability and specific probability can more accurately evaluate the target word segmentation, thereby improving the accuracy of specific type recognition of the target word segmentation.

[0148] In step S330, related words corresponding to adjacent words of the target word are obtained from the preset knowledge graph, and the target word in the word set is replaced with related words.

[0149] Specifically, a pre-defined knowledge graph can be used to store pre-defined word segments, which can be represented based on the knowledge graph. The knowledge graph represents knowledge (i.e., pre-defined word segments) in the form of a graph or network. The knowledge graph consists of multiple nodes, each representing an entity, and the edges between nodes represent relationships between entities. It should be noted that an entity is a word with actual meaning and can be considered as an individual, such as "Marx," "Shenzhen," or "Earth."

[0150] As an optional embodiment, the above method further includes: obtaining at least one sample preset word segmentation in a preset knowledge graph; wherein, at least one sample preset word segmentation corresponds to an adjacent word segmentation of the target word; inputting at least one sample preset word segmentation into a classification model to train the classification model to perform specific type discrimination on at least one sample preset word segmentation and obtain discrimination results; and updating the classification model based on the discrimination results.

[0151] Specifically, at least one sample word segmentation vector has a vector distance less than a preset distance between it and the target word segmentation vector. This vector distance can be either cosine distance or Euclidean distance. Furthermore, the above-mentioned discrimination result can be used to indicate whether the preset word segmentation belongs to a specific type.

[0152] Please see Figure 8 , Figure 8 The illustration shows a schematic diagram of a classification model structure based on model training according to an embodiment of this application. Figure 8 As shown, the GCN model 880 can be used to extract features from the preset word segments "foreign currency", "transaction", "currency exchange" and "exchange rate" of the sample. Based on the feature extraction, the first feature extraction layer 820 in the classification model is trained to improve the feature extraction accuracy of the first feature extraction layer 820. The GCN model 880 is a graph convolutional network.

[0153] Specifically, based on the trained classification model, each word segmentation set [who, can, exchange for a pen, exchange rate, discount] can be sequentially input into the first feature extraction layer 820 through the input layer 810. The first feature extraction layer 820 extracts the feature vectors of [who, can, exchange for a pen, exchange rate, discount] and inputs them into the fully connected layer 830. The fully connected layer 830 can associate the received feature vectors so that the associated feature vectors can fuse some features of other feature vectors, and the association result is input into the second feature extraction layer 840 for further feature extraction. Then, the second feature extraction layer 840 can input the feature extraction into the fully connected layer 850 so that the fully connected layer 850 can further associate the received feature vectors and input the association result into the third feature extraction layer 860 so that the third feature extraction layer 860 can perform further feature extraction and input it into the output layer 870 so that the output layer 870 can predict the specific probability that the target word belongs to a specific type, and determine whether it belongs to a specific type based on the specific probability. For example, the specific probabilities corresponding to [who, can, exchange for a pen, exchange rate, discount] are [0%, 0%, 100%, 0%, 0%], so the judgment results corresponding to [who, can, exchange for a pen, exchange rate, discount] can be [no, no, yes, no, no].

[0154] The first feature extraction layer 820 can be configured as a neural network (RNN), gated recurrent unit (GRU), or long short-term memory (LSTM) for processing sequence data.

[0155] It should be noted that the first feature extraction layer 820, the second feature extraction layer 840, and the third feature extraction layer 860 correspond to different feature extraction parameters. The fully connected layer 830 and the fully connected layer 850 also correspond to different network parameters.

[0156] As can be seen, implementing this optional embodiment can train the classification model in a targeted manner, which is beneficial to improving the recognition accuracy of specific types of target words.

[0157] As an optional embodiment, obtaining related word segments corresponding to adjacent word segments of the target word from a preset knowledge graph includes: comparing the preset knowledge graph with the target word by adjacent word segments to obtain a comparison result; if the comparison result indicates that the adjacent word segments match the preset knowledge graph, then obtaining related word segments corresponding to adjacent word segments of the target word from the preset knowledge graph. The adjacent word segment can be either the word preceding or following the target word segment; this embodiment does not limit the scope of the application.

[0158] Specifically, the pre-defined knowledge graph is compared with the target word segmentation and adjacent word segments to obtain the comparison results. This includes comparing each word group (e.g., a triplet, which is a combination containing three words) in the pre-defined knowledge graph with the adjacent word segments to obtain the comparison results. The comparison results are used to characterize whether the adjacent word segments hit the pre-defined knowledge graph. Among them, the triplet can be stored in the form of subject-verb-object, where the subject (e.g., pig), the verb (e.g., is), and the object (e.g., animal) are three words. These three words have a word order relationship, and the subject and object can correspond to the same part of speech.

[0159] Based on this, if there are multiple related word segments, the adjacent word segments in the word segment set are replaced with related word segments, including: randomly selecting one of the related word segments as a replacement word; and replacing adjacent word segments in the word segment set with the replacement word. For example, if the related word segments corresponding to the hit word "exchange rate" in the preset knowledge graph that are consistent with the adjacent word "exchange rate" include "HKD", "foreign currency", "transaction", and "currency exchange", then the related word "HKD" is randomly selected, and "pen" in [who, can, exchange, pen, exchange rate] is replaced with [who, can, exchange, HKD, exchange rate]. Among them, "HKD" and "foreign currency" can be in the same triplet; "exchange rate" and "foreign currency" can be in the same triplet; "exchange rate" and "transaction" can be in the same triplet; and "exchange rate" and "currency exchange" can be in the same triplet.

[0160] As can be seen, implementing this optional embodiment can help obtain a score for evaluating the target word by replacing the target word, thereby helping to identify whether the target word is a specific type of word from multiple dimensions and improving the recognition accuracy.

[0161] In one exemplary embodiment of this application, obtaining related words corresponding to adjacent words of the target word from a preset knowledge graph includes: determining the word group in which the adjacent word is located from the preset knowledge graph; selecting word groups whose parts of speech are consistent with those of the adjacent word as related words; wherein, the related words are random word groups in the word group or word groups with the highest call frequency.

[0162] Specifically, the word groups of adjacent word segments are determined from the preset knowledge graph, including: determining the word group of the word preceding the target word and the word group of the word following the target word from the preset knowledge graph; among them, the word segment with the highest call frequency can be understood as the popular word segment.

[0163] In addition, selecting a word from the phrase that has the same part of speech as the adjacent word as a related word includes selecting at least one word from the phrase that has the same part of speech (e.g., noun) as the related word.

[0164] As can be seen, by implementing this optional embodiment, suitable replacement words can be determined based on the selection of relevant word segments, and the target word in the word segmentation set can be replaced to obtain an evaluation of the target word in another dimension. This is beneficial for identifying specific types of target word segments based on multi-dimensional evaluation and improving recognition accuracy.

[0165] In one exemplary embodiment of this application, if the number of related words is greater than 1, the parameter determination unit determines the reference score of the related words based on each word in the word segmentation set, including: determining the reference score of each related word after it is replaced in the word segmentation set, and obtaining a reference score set; calculating the weighted average of the reference score set as the reference score of the related words.

[0166] The calculation of the weighted average of the reference score set as the reference score for the relevant word segmentation includes: obtaining the weight value of each reference score in the reference score set, and calculating the weighted average of the reference score set based on the weight value as the reference score for the relevant word segmentation.

[0167] As can be seen, implementing this optional embodiment can integrate the scores of all relevant word segments, which is beneficial for identifying specific types of target word segments based on multi-dimensional evaluation and improving recognition accuracy.

[0168] In step S340, a reference score for each word in the word segmentation set is determined. The reference score for the relevant word is used to evaluate the target word in a specific way.

[0169] Specifically, the reference scores for related words are determined based on each word in the word segmentation set, including: determining the reference scores for related words based on their arrangement position in the word segmentation set and the reference scores for each word in the word segmentation set. The reference scores for related words are also obtained based on prediction and classification models, and their acquisition process is the same as that for the reference scores of the target words.

[0170] In step S350, the target word is identified as a specific type based on the reference score of the target word and the reference score of related words.

[0171] Specifically, the specific type can be a designated word segmentation type, such as a rewritten word type for sensitive words, a restricted word type, etc., but this application embodiment does not limit it.

[0172] In addition, after identifying the target word based on the reference score of the target word and the reference score of related words, the above method may also include: if the target word is identified as a specific type of word segmentation, then each character in the target word is converted into a specific character (e.g., *) according to the number of characters in the target word, and / or, outputting a prompt message to indicate that there are sensitive words in the text (e.g., there are sensitive words in your output text, and the sensitive words cannot be displayed normally).

[0173] As an optional embodiment, specific type identification of the target word is performed based on the reference score of the target word and the reference score of related words, including: determining the difference between the reference score of the target word and the reference score of related words; and performing specific type identification of the target word based on the difference and the reference score of the target word.

[0174] Specifically, determining the difference between the reference score of the target word and the reference score of the related word includes: calculating the difference s1 = |s2-s3| between the reference score s2 of the target word and the reference score s3 of the related word.

[0175] As can be seen, implementing this optional embodiment can improve the accuracy of specific type recognition by performing specific type recognition on target words based on the difference between the reference score of the target word and the reference score of related words.

[0176] As an optional embodiment, the target word is identified as a specific type based on the difference and the reference score of the target word, including: calculating the weighted sum of the difference and the reference score of the target word according to the preset weight value; if the weighted sum is detected to be greater than or equal to the preset threshold value, the target word is determined to belong to a specific type; wherein, the specific type is used to specifically refer to the target word being a rewritten word against preset sensitive words.

[0177] Specifically, the preset weight values ​​include the weight values ​​corresponding to the difference and the reference score of the target word. The weighted sum of the difference and the reference score of the target word is calculated based on the preset weight values, including: the weighted sum of the difference s1 and the reference score of the target word s2 (a1*s1+a2*s2) is calculated based on the weight values ​​a1 and a2 corresponding to the difference and the reference score of the target word, respectively. a1, s1, a2, and s2 are all constants.

[0178] As can be seen, implementing this optional embodiment can reasonably allocate the proportion of the difference and the reference score of the target word in the weighted sum based on the weights, thereby helping to improve the recognition accuracy of specific types.

[0179] Please see Figure 9 , Figure 9 A flowchart illustrating a method for recognizing specific types of word segmentation according to an embodiment of this application is shown schematically. Figure 9As shown, the identification method for specific type word segmentation includes steps S900 to S980.

[0180] Step S900: Segment the sample text in the sample conversation character by character to obtain a character set for constructing the sample text. Combine adjacent characters in the character set according to the set number of adjacent characters and the positional relationship of characters in the sample text to obtain a phrase set corresponding to the sample text.

[0181] Step S910: Obtain the set of all phrases in the sample session to get the total phrase set, and calculate the frequency of each phrase in the total phrase set based on the total number of phrases in the total phrase set. Determine the phrases with a frequency higher than the frequency threshold as specific phrases.

[0182] Step S920: Perform word segmentation on the text to be analyzed to obtain a reference word segmentation set.

[0183] Step S930: Merge the words to be merged in the reference word segmentation set based on a specific phrase to achieve word reorganization and obtain a word segmentation set; wherein, the words to be merged are the words in the reference word segmentation set that hit the same specific phrase, and the number of words in the word segmentation set is less than or equal to the number of words in the reference word segmentation set.

[0184] Step S940: Obtain at least one sample preset word segmentation in the preset knowledge graph, wherein at least one sample preset word segmentation corresponds to the adjacent word segmentation of the target word, and then input the at least one sample preset word segmentation into the classification model to train the classification model to perform specific type discrimination on the at least one sample preset word segmentation, obtain the discrimination result, and update the classification model according to the discrimination result.

[0185] Step S950: Based on the prediction model, predict the occurrence probability of the target word according to the preceding word adjacent to the target word, predict the specific probability of the target word belonging to a specific type based on the classification model, and determine the reference score of the target word based on the occurrence probability and the specific probability.

[0186] Step S960: Compare the adjacent segments of the target segment with the preset knowledge graph to obtain the comparison result. If the comparison result indicates that the adjacent segment matches the preset knowledge graph, then obtain the related segments corresponding to the adjacent segments of the target segment from the preset knowledge graph, and replace the target segment in the segment set with the related segments.

[0187] Step S970: Determine the reference score of the relevant segment based on each segment in the segmentation set. The reference score of the relevant segment is used to evaluate the target segment in a specific way.

[0188] Step S980: Determine the difference between the reference score of the target word and the reference score of related words. Calculate the weighted sum of the difference and the reference score of the target word based on the preset weight value. If the weighted sum is detected to be greater than or equal to the preset threshold value, the target word is determined to belong to a specific type. The specific type is used to specifically refer to the target word being a rewritten word targeting preset sensitive words.

[0189] It should be noted that steps S900 to S980 are the same as... Figure 3 For the specific implementation details of steps S900 to S980, please refer to the examples shown. Figure 3 The steps and their embodiments shown are not repeated here.

[0190] It is evident that implementation Figure 9 The method described herein can obtain specific type evaluations of word segmentation under multiple dimensions through multi-dimensional analysis of word segmentation in text. This allows for the combination of multi-dimensional evaluations to identify specific types of word segmentation, overcoming the limitations of existing word lists. Furthermore, it can leverage artificial intelligence technology to achieve multi-dimensional analysis of word segmentation in text, improving the recognition accuracy for specific types of word segmentation. In addition, it can replace target word segmentation based on preset word segmentation in a pre-defined knowledge graph (e.g., a knowledge graph itself), using the reference scores before and after replacement as a multi-dimensional evaluation of the target word segmentation, thereby achieving the mining of specific types of word segmentation. Since this application does not require constructing a word list for specific types of word segmentation, it avoids the resource consumption associated with such word lists, thus improving the utilization of computer resources.

[0191] Furthermore, this example embodiment also provides a recognition device for word segmentation of a specific type. (See reference) Figure 10 As shown, the recognition device 1000 for specific type word segmentation may include: a word segmentation acquisition unit 1001, a parameter determination unit 1002, a word segmentation replacement unit 1003, and a specific type word segmentation recognition unit 1004, wherein:

[0192] The word segmentation acquisition unit 1001 is used to acquire the word segmentation set of the text to be analyzed;

[0193] The parameter determination unit 1002 is used to determine the reference score of the target word in the word segmentation set based on each word in the word segmentation set. The reference score of the target word is used to evaluate the target word in a specific type.

[0194] The word segmentation and replacement unit 1003 is used to obtain related words corresponding to adjacent words of the target word from a preset knowledge graph, and replace the target word in the word segmentation set with related words;

[0195] The parameter determination unit 1002 is also used to determine the reference score of the relevant segment based on each segment in the segmentation set. The reference score of the relevant segment is used to evaluate the target segment in a specific type.

[0196] The specific type word segmentation recognition unit 1004 is used to perform specific type recognition on the target word based on the reference score of the target word and the reference score of related words.

[0197] The target word is any word in the word segmentation set.

[0198] It is evident that implementation Figure 10 The device shown can obtain specific type evaluations of word segmentation in multiple dimensions through multi-dimensional analysis of word segmentation in text. This allows for the combined use of multi-dimensional evaluations to identify specific types of word segmentation, overcoming the limitations of existing word lists. Furthermore, it can leverage artificial intelligence technology to perform multi-dimensional analysis of word segmentation in text, improving the recognition accuracy for specific types of word segmentation. In addition, it can replace target word segmentation based on preset word segmentation in a preset knowledge graph (e.g., a knowledge graph itself), using the reference scores before and after replacement as a multi-dimensional evaluation of the target word segmentation, thereby achieving the mining of specific types of word segmentation. Since this application does not require constructing a word list for specific types of word segmentation, it avoids the resource consumption associated with such word lists, thus improving the utilization of computer resources.

[0199] In one exemplary embodiment of this application, the word segmentation acquisition unit 1001 acquires a set of word segments of the text to be analyzed, including:

[0200] Extract specific phrases that appear more frequently than a frequency threshold in sample conversations;

[0201] The obtained text to be analyzed is segmented into words to obtain a reference segmentation set;

[0202] The reference word segmentation set is reorganized based on specific phrases to obtain the word segmentation set.

[0203] As can be seen, implementing this optional embodiment can extract specific phrases based on real user conversations and modify the word segmentation set according to the specific phrases, thereby improving the accuracy of word segmentation and thus improving the recognition accuracy of specific types of word segmentation.

[0204] In one exemplary embodiment of this application, the above-described apparatus further includes:

[0205] The session reading unit (not shown) is used to read at least one set of historical sessions before the word segmentation acquisition unit 1001 extracts specific phrases that appear in the sample sessions with a frequency higher than the frequency threshold, and to filter sample sessions from at least one set of historical sessions according to the number of texts; wherein the number of texts contained in the sample sessions is greater than the preset number of texts.

[0206] As can be seen, implementing this optional embodiment can filter historical sessions to obtain valid sample sessions, which can help improve the recognition accuracy of specific phrases in real user sessions and reduce the waste of computing resources caused by invalid sample sessions.

[0207] In one exemplary embodiment of this application, the word segmentation acquisition unit 1001 extracts specific phrases from sample sessions that appear more frequently than a frequency threshold, including:

[0208] The sample text in the sample conversation is segmented character by character to obtain the set of characters used to construct the sample text;

[0209] The adjacent characters in the character set are combined according to the set number of adjacent characters and the positional relationship of characters in the sample text to obtain the phrase set corresponding to the sample text.

[0210] Obtain the set of all phrases in the sample session to get the total phrase set;

[0211] The frequency of occurrence of each phrase in the total phrase set is calculated based on the total number of phrases in the total phrase set.

[0212] Phrases that appear more frequently than a frequency threshold are identified as specific phrases.

[0213] As can be seen, implementing this optional embodiment can use real user sessions as sample sessions for specific phrase segmentation, thereby using specific phrases as the basis for optimizing the segmentation set, improving segmentation accuracy, and reducing recognition errors of specific types of segmentation due to segmentation errors.

[0214] In one exemplary embodiment of this application, the word segmentation acquisition unit 1001 performs word segmentation processing on the acquired text to be analyzed to obtain a reference word segmentation set, including:

[0215] Select the text to be analyzed according to the order in which the text is published in the current session;

[0216] The text to be analyzed is segmented into words to obtain a reference segmentation set.

[0217] As can be seen, by implementing this optional embodiment, the corresponding word segmentation set can be obtained by segmenting the text to be analyzed, and then the words in the set can be analyzed sequentially to identify specific types of words.

[0218] In one exemplary embodiment of this application, the word segmentation acquisition unit 1001 performs word segmentation and reorganization on a reference word segmentation set based on a specific phrase to obtain a word segmentation set, including:

[0219] Based on specific phrases, the words to be merged in the reference word segmentation set are merged to achieve word segmentation reorganization and obtain a word segmentation set;

[0220] Among them, the word segments to be merged are the word segments in the reference word segmentation set that hit the same specific phrase, and the number of word segments in the word segmentation set is less than or equal to the number of word segments in the reference word segmentation set.

[0221] As can be seen, implementing this optional embodiment can avoid word segmentation errors based on word segmentation and recombination, thereby improving the recognition accuracy for specific types of word segmentation.

[0222] In one exemplary embodiment of this application, the word segmentation replacement unit 1003 obtains related words corresponding to adjacent words of the target word from a preset knowledge graph, including:

[0223] The preset knowledge graph is compared with the target word segmentation and adjacent word segments to obtain the comparison results;

[0224] If the comparison result is used to indicate that adjacent word segments hit the preset knowledge graph, then the relevant word segments corresponding to the adjacent word segments of the target word are obtained from the preset knowledge graph.

[0225] As can be seen, implementing this optional embodiment can help obtain a score for evaluating the target word by replacing the target word, thereby helping to identify whether the target word is a specific type of word from multiple dimensions and improving the recognition accuracy.

[0226] In one exemplary embodiment of this application, the word segmentation replacement unit 1003 obtains related words corresponding to adjacent words of the target word from a preset knowledge graph, including:

[0227] Determine the word groups where adjacent word segments are located from a pre-defined knowledge graph;

[0228] Select the word segment whose part of speech is consistent with the adjacent word segment from the word group as the related word segment; among them, the related word segment is the random word segment in the word group or the word segment with the highest call frequency in the word group.

[0229] As can be seen, by implementing this optional embodiment, suitable replacement words can be determined based on the selection of relevant word segments, and the target word in the word segmentation set can be replaced to obtain an evaluation of the target word in another dimension. This is beneficial for identifying specific types of target word segments based on multi-dimensional evaluation and improving recognition accuracy.

[0230] In one exemplary embodiment of this application, if the number of related words is greater than 1, the parameter determination unit determines a reference score for the related words based on each word in the word segmentation set, including:

[0231] Determine a reference score for each relevant word segment after it has been replaced in the word segment set, and obtain a reference score set;

[0232] Calculate the weighted average of the reference score set as the reference score for the relevant word segmentation.

[0233] As can be seen, implementing this optional embodiment can integrate the scores of all relevant word segments, which is beneficial for identifying specific types of target word segments based on multi-dimensional evaluation and improving recognition accuracy.

[0234] In one exemplary embodiment of this application, the specific type word segmentation recognition unit 1004 performs specific type recognition on the target word based on the reference score of the target word and the reference scores of related words, including:

[0235] Determine the difference between the reference score of the target word segmentation and the reference score of related words;

[0236] The target word is identified by specific type based on the difference and the reference score of the target word.

[0237] As can be seen, implementing this optional embodiment can improve the accuracy of specific type recognition by performing specific type recognition on target words based on the difference between the reference score of the target word and the reference score of related words.

[0238] In one exemplary embodiment of this application, the specific type word segmentation recognition unit 1004 performs specific type recognition on the target word based on the difference and the reference score of the target word, including:

[0239] The weighted sum of the difference and the reference score of the target word segmentation is calculated based on the preset weight values.

[0240] If the weighted sum is detected to be greater than or equal to a preset threshold, the target word segmentation is determined to belong to a specific type; where the specific type is used to specifically refer to the target word segmentation as a rewritten word against preset sensitive words.

[0241] As can be seen, implementing this optional embodiment can reasonably allocate the proportion of the difference and the reference score of the target word in the weighted sum based on the weights, thereby helping to improve the recognition accuracy of specific types.

[0242] In one exemplary embodiment of this application, the parameter determination unit 1002 determines a reference score for the target word in the word segmentation set based on each word segmentation in the word segmentation set, including:

[0243] The prediction model predicts the probability of the target word appearing based on the preceding word adjacent to the target word;

[0244] Predict the specific probability that the target word segment belongs to a specific type based on a classification model;

[0245] The reference score for target word segmentation is determined based on the probability of occurrence and a specific probability.

[0246] As can be seen, by implementing this optional embodiment, the target word segmentation can be probabilistically predicted through prediction and classification models. The reference score of the target word segmentation determined based on the above occurrence probability and specific probability can more accurately evaluate the target word segmentation, thereby improving the accuracy of specific type recognition of the target word segmentation.

[0247] In one exemplary embodiment of this application, the above-described apparatus further includes:

[0248] The model training unit (not shown) is used to obtain at least one sample pre-defined word segmentation in the pre-defined knowledge graph; wherein, at least one sample pre-defined word segmentation corresponds to the adjacent word segmentation of the target word; at least one sample pre-defined word segmentation is input into the classification model to train the classification model to perform specific type discrimination on at least one sample pre-defined word segmentation and obtain the discrimination result; the classification model is updated according to the discrimination result.

[0249] As can be seen, implementing this optional embodiment can train the classification model in a targeted manner, which is beneficial to improving the recognition accuracy of specific types of target words.

[0250] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0251] Since the functional modules of the recognition device for specific type word segmentation in the example embodiments of this application correspond to the steps of the recognition method for specific type word segmentation described above, for details not disclosed in the device embodiments of this application, please refer to the embodiments of the recognition method for specific type word segmentation described above in this application.

[0252] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0253] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0254] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0255] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0256] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0257] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for recognizing specific types of word segmentation, characterized in that, include: Obtain the word segmentation set of the text to be analyzed; A reference score for the target word in the word segmentation set is determined based on each word in the word segmentation set; The reference score for the target word segmentation is used to evaluate the target word segmentation in a specific way; The preset knowledge graph is compared with the adjacent words of the target word to obtain the comparison result. If the comparison result indicates that the adjacent word hits the preset knowledge graph, the relevant words corresponding to the adjacent word are obtained from the preset knowledge graph, and the target word in the word set is replaced with the relevant words. Based on the prediction model, the occurrence probability of the related word is predicted according to the preceding word adjacent to the target word, and the specific probability of the related word belonging to the specific type is predicted based on the classification model. A reference score of the related word is determined based on the occurrence probability and the specific probability. The reference score of the related word is used to evaluate the target word for a specific type. The target word is identified by a specific type based on the reference score of the target word and the reference score of the related words.

2. The method according to claim 1, characterized in that, Obtain the word segmentation set of the text to be analyzed, including: Extract specific phrases that appear more frequently than a frequency threshold in sample conversations; The obtained text to be analyzed is segmented into words to obtain a reference segmentation set; The reference word segmentation set is reorganized based on the specific phrase to obtain the word segmentation set.

3. The method according to claim 2, characterized in that, Before extracting specific phrases that appear more frequently than a frequency threshold in sample sessions, the method further includes: Read at least one set of historical sessions, and filter the sample sessions from the at least one set of historical sessions according to the number of texts; wherein the number of texts contained in the sample sessions is greater than a preset number of texts.

4. The method according to claim 2, characterized in that, Extract specific phrases from sample conversations that appear more frequently than a frequency threshold, including: The sample text in the sample session is segmented character by character to obtain a set of characters used to constitute the sample text; The adjacent characters in the character set are combined according to the set number of adjacent characters and the positional relationship of characters in the sample text to obtain the phrase set corresponding to the sample text; Obtain the set of all phrases in the sample session to obtain the total phrase set; The frequency of occurrence of each phrase in the total phrase set is calculated based on the total number of phrases in the total phrase set; Phrases that appear more frequently than the frequency threshold are identified as the specific phrases.

5. The method according to claim 2, characterized in that, The obtained text to be analyzed is segmented into words to obtain a reference segmentation set, including: The text to be analyzed is selected according to the order in which the text is published in the current session; The text to be analyzed is segmented into words to obtain the reference segmentation set.

6. The method according to claim 2, characterized in that, Based on the specific phrase, the reference word segmentation set is segmented and reorganized to obtain the word segmentation set, which includes: Based on the specific phrase, the words to be merged in the reference word segmentation set are merged to achieve word segmentation reorganization and obtain the word segmentation set; The word segments to be merged are those that hit the same specific phrase in the reference word segmentation set, and the number of word segments in the word segmentation set is less than or equal to the number of word segments in the reference word segmentation set.

7. The method according to claim 1, characterized in that, Obtaining related word segments corresponding to the adjacent word segments from the preset knowledge graph includes: The word groups in which the adjacent word segments are located are determined from the preset knowledge graph; The word segment whose part of speech is consistent with the adjacent word segment is selected from the word group as the related word segment.

8. The method according to claim 1, characterized in that, If the number of related words is greater than 1, a reference score for the related words is determined based on each word in the word segmentation set, including: Determine a reference score for each relevant word segment after it has been replaced in the word segment set, and obtain a reference score set; The weighted average of the reference score set is calculated as the reference score for the relevant word segmentation.

9. The method according to claim 1, characterized in that, Based on the reference scores of the target word segmentation and the reference scores of the related words, specific type identification is performed on the target word segmentation, including: Determine the difference between the reference score of the target word segment and the reference score of the related word segment; The target word is identified as having a specific type based on the difference and the reference score of the target word.

10. The method according to claim 1, characterized in that, The method further includes: Obtain at least one sample preset word segmentation from the preset knowledge graph; wherein, the at least one sample preset word segmentation corresponds to the adjacent word segmentation of the target word; The at least one sample pre-defined word segment is input into the classification model to train the classification model to perform specific type discrimination on the at least one sample pre-defined word segment and obtain the discrimination result; The classification model is updated based on the discrimination result.

11. A recognition device for word segmentation of a specific type, characterized in that, include: The word segmentation acquisition unit is used to acquire the word segmentation set of the text to be analyzed; The parameter determination unit is used to determine the reference score of the target segment in the word segmentation set based on each segment in the word segmentation set, and the reference score of the target segment is used to evaluate the target segment in a specific type. The word segmentation and replacement unit is used to compare the adjacent words of the target word with the preset knowledge graph to obtain the comparison result. If the comparison result indicates that the adjacent word hits the preset knowledge graph, then the relevant words corresponding to the adjacent word are obtained from the preset knowledge graph, and the target word in the word set is replaced with the relevant words. The parameter determination unit is further configured to predict the occurrence probability of the related word based on the preceding word adjacent to the target word using a prediction model, and predict the specific probability of the related word belonging to the specific type based on a classification model, and determine a reference score for the related word based on the occurrence probability and the specific probability; the reference score for the related word is used to evaluate the target word for a specific type. A specific type word segmentation recognition unit is used to perform specific type recognition on the target word based on the reference score of the target word and the reference score of the related words.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-10.

13. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-10 by executing the executable instructions.

14. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to cause the computer device to perform the method according to any one of claims 1-10.

Images