Text detection method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a text detection method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: determining a first word and a first affix in a first text; determining a first index interval of the first word and a second index interval of the first affix based on a first index of a character in the first text in the first text; determining a position relationship between the first word and the first affix based on the first index interval and the second index interval; under the condition that the position relation represents that the first word is adjacent to the first affix, the first affix is verified based on a correct affix set for the first word, and a detection result of the first text is obtained. According to the invention, the precision and reliability of text detection can be improved.

Description

Technical Field

[0001] The present application relates to text processing technology, and in particular to a text detection method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Text detection technology is widely used in fields such as spelling checking, grammar correction, content review, machine translation, and educational assessment. However, when dealing with complex language structures, the file detection methods in related technologies have low accuracy and are prone to misjudgment, affecting the overall detection effect. Summary of the Invention

[0003] Embodiments of the present application provide a text detection method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy and reliability of text detection.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides a text detection method, the method comprising:

[0006] determining a first word and a first affix in a first text;

[0007] Determining a first index interval of the first word and a second index interval of the first affix based on a first index of a character in the first text;

[0008] Determining a positional relationship between the first word and the first affix based on the first index interval and the second index interval;

[0009] When the positional relationship indicates that the first word is adjacent to the first affix, the first affix is ​​verified based on a correct affix set for the first word to obtain a detection result of the first text.

[0010] An embodiment of the present application provides a text detection device, including:

[0011] A word determination module, configured to determine a first word and a first affix in a first text;

[0012] An index determination module, configured to determine a first index interval of the first word and a second index interval of the first affix based on a first index of a character in the first text;

[0013] a position determination module, configured to determine a positional relationship between the first word and the first affix based on the first index interval and the second index interval;

[0014] The detection module is configured to verify the first affix based on a correct affix set for the first word when the positional relationship indicates that the first word is adjacent to the first affix, so as to obtain a detection result of the first text.

[0015] An embodiment of the present application provides an electronic device, comprising:

[0016] a memory for storing computer-executable instructions or computer programs;

[0017] The processor is configured to implement the text detection method provided in the embodiment of the present application when executing the computer-executable instructions or computer program stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the text detection method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the text detection method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] By determining the first index interval of the first word and the second index interval of the first affix in the first text, and analyzing the positional relationship between the first word and the first affix based on the first index interval and the second index interval, the association between the word and the affix can be identified more accurately. When the first word is adjacent to the first affix, verification is performed based on a preset correct affix standard, thereby obtaining the detection result of the first text by detecting whether the affix adjacent to the first word is wrong. Compared with the related art that only detects whether there are keywords in the text to determine whether the text is abnormal, the present application adds affix detection, thereby improving the accuracy and reliability of text detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a structural diagram of a text detection system provided in an embodiment of the present application;

[0023] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0024] Figure 3 This is a flowchart of the text detection method provided in the embodiment of the present application. Figure 1 ;

[0025] Figure 4 This is a flowchart of the text detection method provided in the embodiment of the present application. Figure 2 ;

[0026] Figure 5 This is a schematic diagram of the structure of the hash tree provided in the embodiment of the present application;

[0027] Figure 6 This is a flowchart of the text detection method provided in the embodiment of the present application. Figure 3 ;

[0028] Figure 7 This is a flowchart of the text detection method provided in the embodiment of the present application. Figure 4 ;

[0029] Figure 8 This is a schematic diagram of a node tree provided in an embodiment of the present application;

[0030] Figure 9 This is a flowchart of the text detection method provided in the embodiment of the present application. Figure 5 ;

[0031] Figure 10 is an index representation scheme of a first index and a second index of a character in a first text provided by an embodiment of the present application;

[0032] Figure 11 This is a flowchart of the text detection method provided in the embodiment of the present application. Figure 6 ;

[0033] Figure 12 Schematic diagram of the text detection method provided in the embodiment of the present application;

[0034] Figure 13 is a schematic diagram of a text node list provided in an embodiment of the present application;

[0035] Figure 14 This is a schematic diagram of assigning attribute values ​​to text nodes provided by an embodiment of the present application;

[0036] Figure 15 This is a schematic diagram of traversing the hash tree provided by the embodiment of the present application. Figure 1 ;

[0037] Figure 16 This is a schematic diagram of traversing the hash tree provided by the embodiment of the present application. Figure 2 ;

[0038] Figure 17 This is a schematic diagram of standard detection words obtained by traversing the hash tree provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0041] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0042] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0043] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0044] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0045] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0046] 1) Text detection: This refers to the analysis and processing of text content using computer technology to identify and correct errors such as spelling errors, grammatical errors, and formatting errors. Common text detection applications include spell checking, grammar correction, and content review.

[0047] 2) Affix: This refers to a component attached to the basic part (root) of a word to change the meaning or grammatical function of the word. Affixes are divided into two types: prefixes and suffixes. In the embodiment of the present application, an affix refers to a preset vocabulary related to the word to be tested or a context vocabulary.

[0048] 3) Index: This refers to a location identifier within a data structure used to quickly find data. In text processing, an index typically refers to the position of a character or word within a text. For example, in the string "hello," the character "h" has index 0, and the character "e" has index 1.

[0049] 4) Tree structure: A nonlinear data structure consisting of nodes and edges, where each node can have zero or more child nodes but only one parent node (except the root node). Tree structures are often used to represent hierarchical relationships, such as file systems and organizational structures.

[0050] 5) Hash Tree: A data structure that combines a hash table and a tree structure for efficient data storage and retrieval. A hash tree maps data to tree nodes using a hash function. Each node can store multiple data items or point to other child nodes.

[0051] 6) Traversal: This refers to accessing every element in a data structure in a certain order. In tree structures, common traversal methods include pre-order traversal, in-order traversal, and post-order traversal. In text processing, traversal usually refers to accessing each character or word in the text one by one.

[0052] 8) Web Document: A document transmitted over the Internet and displayed in a browser, typically written in Hypertext Markup Language (HTML). A web document contains multiple elements such as text, images, and links, used to display information and provide interactive functionality.

[0053] 9) Node tree: This refers to a tree structure that represents each element in a document as a node. In an HTML document, each tag, attribute, and text content can be considered a node, and the relationships between nodes form a tree. Node trees are often used to parse and manipulate web documents.

[0054] 10) Tags: are the basic elements in HTML documents, used to define the structure and content of the document. Tags are usually surrounded by angle brackets < and >, for example Represents a paragraph tag, Represents a hyperlink tag.

[0055] 11) Lexicon: A lexicon is a database or file that stores a large number of words and their associated information. Lexicons are commonly used in natural language processing tasks such as spell checking, part-of-speech tagging, and sentiment analysis. Lexicons can include information such as the correct spelling, part of speech, meaning, and affixes of a word.

[0056] Embodiments of the present application provide a text detection method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy and reliability of text detection.

[0057] In the text detection method provided in the embodiment of the present application, first, a first word and a first affix in a first text are determined; then, based on the first index of the characters in the first text in the first text, a first index interval of the first word and a second index interval of the first affix are determined; then, based on the first index interval and the second index interval, a positional relationship between the first word and the first affix is ​​determined; finally, when the positional relationship indicates that the first word and the first affix are adjacent, the first affix is ​​verified based on the correct affix set for the first word to obtain a detection result of the first text.

[0058] The exemplary application of the text detection device provided by the embodiment of the present application is described below. The text detection device is an electronic device for implementing a text detection method. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application. Below, the exemplary application of the text detection device when it is implemented as a terminal or a server will be described.

[0059] See also Figure 1 , Figure 1 It is the framework diagram of the text detection system that the embodiment of the application provides, for executing the text detection operation, a text detection application can be provided, for example, this text detection application can be an application dedicated to text detection, or it can be a functional module (such as the text detection module in the web application, etc.) in other applications. The text detection system 100 of the embodiment of the application at least includes terminal 400, network 300 and server 200, wherein server 200 is the server of text detection application. Server 200 can constitute the text detection equipment of the embodiment of the application, namely realizes the text detection method of the embodiment of the application by server 200. Terminal 400 connects server 200 through network 300, and network 300 can be wide area network or local area network, or a combination of the two.

[0060] See also Figure 1 , a user can perform interactive operations on the client side of the text detection application through terminal 400. Such interactive operations may include, for example, inputting a first text, performing a detection operation on the first text, and the like. After receiving the user's interactive operation, the client side encapsulates the first text into a first text detection request and sends the first text detection request to server 200 via network 300. After receiving the first text detection request, server 200 responds to the first text detection request sent by the terminal and determines the first word and the first affix in the first text. Based on the first index of the characters in the first text, server 200 determines the first index interval of the first word and the second index interval of the first affix. Based on the first index interval and the second index interval, server 200 determines the positional relationship between the first word and the first affix. If the positional relationship indicates that the first word and the first affix are adjacent, server 200 verifies the first affix based on the correct affix set for the first word to obtain a detection result for the first text. If the detection result of the first text indicates that the first word is an abnormal word, server 200 returns a notification message to terminal 400 indicating that the first word is an abnormal word. After receiving the notification message, the terminal 400 marks the first word on the current interface.

[0061] In some embodiments, the terminal 400 may also perform the text detection method of the embodiment of the present application, that is, after the user performs an interactive operation on the client of the text detection application through the terminal 400, the terminal 400 determines the first word and the first affix in the first text in response to the interactive operation; the terminal 400 determines the first index interval of the first word and the second index interval of the first affix based on the first index of the character in the first text; the terminal 400 determines the positional relationship between the first word and the first affix based on the first index interval and the second index interval; the terminal 400 verifies the first affix based on the correct affix set for the first word when the positional relationship indicates that the first word and the first affix are adjacent to each other, and obtains the detection result of the first text. When the detection result of the first text is that the first word is an abnormal word, the terminal 400 marks the first word on the current interface.

[0062] In the scenario of checking whether the knowledge content entered into a web page complies with the standards, if the knowledge user obtains incorrect knowledge, it will directly affect the user's decision-making and greatly reduce the trust of the knowledge manager in the team. Therefore, both knowledge managers and knowledge users need to check whether the knowledge entry is correct and complies with the standards. The user can complete the entry of knowledge content into the web page through terminal 400 to obtain a web page document. The web page document is converted into a node tree through terminal 400, and a first text is determined based on the text content of at least one text node in the node tree; the terminal 400 encapsulates the first text into a text detection request and sends the text detection request to server 200 via network 300. After receiving the text detection request, server 200 determines a first word and a first affix in the first text; based on the first index of the characters in the first text in the first text, determines a first index interval of the first word and a second index interval of the first affix; based on the first index interval and the second index interval, determines the positional relationship between the first word and the first affix; if the positional relationship indicates that the first word and the first affix are adjacent, the first affix is ​​verified based on the correct affix set for the first word to obtain a detection result for the first text. When the detection result indicates that the first word is an abnormal word, a notification message indicating that the first word is an abnormal word may be sent to the terminal 400. The terminal 400 marks and displays the first word on the webpage.

[0063] See also Figure 2 , Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0064] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0066] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0067] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0068] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0069] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0070] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0071] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0072] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0073] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 A text detection device 455 stored in memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a word determination module 4551, an index determination module 4552, a position determination module 4553, and a detection module 4554. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0074] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the text detection method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0075] The following describes the text detection method provided by the embodiment of the present application. As mentioned above, the electronic device implementing the text detection method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0076] It should be noted that, based on their understanding of the following, those skilled in the art can apply the text detection method provided in the embodiments of the present application to a variety of scenarios, such as: spelling checking scenario, in a text editor or online writing platform, automatic detection and correction of spelling errors; grammar correction scenario, in language learning software or document processing tools, identification and correction of grammatical errors; content review scenario, in social media platforms or forums, automatic detection and filtering of inappropriate content; machine translation scenario, in translation tools, detection and correction of affix errors in the source language to improve translation quality; education assessment scenario, in online education platforms, assistance to teachers in correcting students' essays and providing detailed error feedback; search engine optimization scenario, in content management systems, detection and optimization of grammar and spelling of web page content to improve search engine rankings; intelligent customer service scenario, in chatbots or virtual assistant systems, ensuring that the text content of replies is accurate; document proofreading scenario, in the publishing and editing industries, automatic detection and proofreading of affix errors in documents to improve document quality.

[0077] See also Figure 3 , Figure 3 This is a flowchart of the text detection method provided in the embodiment of the present application. Figure 1 , the following will be combined Figure 3 The steps shown are explained as Figure 3 As shown, the text detection method is described by taking the execution subject of the server as an example. The method includes the following steps 101 to 104:

[0078] In step 101 , a first word and a first affix in a first text are determined.

[0079] Here, the first text refers to the original text content that needs to be detected. The first text can be an article, a paragraph, a sentence, or any other form of text data. During the text detection process, the first text is the input data, and the text detection system will analyze and process the first text. The first word refers to a specific word that needs to be detected in the first text. A word consists of one or more characters and can be used independently or combined with other words to form a sentence. The first affix refers to a contextual vocabulary that may be closely related to a first word and has a semantically complementary or limiting effect on the first word.

[0080] Multiple words to be detected and the affixes corresponding to each word can be preset in advance. The affixes corresponding to a word refer to the vocabulary adjacent to the word. Based on the multiple words to be detected and the multiple affixes, the word to be detected appearing in the first text is determined to be the first word, and the affix to be detected appearing in the first text is determined to be the first affix. The number of first words in the first text can be one or more. The number of first affixes in the first text can be one or more.

[0081] For example, the pre-set words to be tested include "in advance," "early repayment," "beware of fraud," and "warm reminder." The suffixes to be tested include "settlement," "all," and "select." For example, if the first text is "early settlement refers to the user's choice to repay part or all of the balance early within the borrowed resource period. The bank allows early settlement, but only after one year," the first words in the first text are "in advance," "early repayment," and the first suffixes are "settlement," "all," and "select."

[0082] In some embodiments, see Figure 4 , Figure 4 It is shown that determining the first word and the first affix in the first text in step 101 can be achieved by the following steps 1011 to 1012:

[0083] In step 1011 , the tree structure is traversed based on each character in the first text.

[0084] Here, the first text includes at least one character. First, a tree structure is pre-constructed to represent a plurality of preset words and affixes to be detected. The tree structure can be a lexical analysis tree (such as a dependency syntax tree), in which each node represents a character. Starting from the first character of the first text, each character is traversed step by step, and each character is matched with a node in the tree structure.

[0085] In the embodiment of the present application, the tree structure can be a hash tree. All preset words and affixes to be detected are obtained, and the first character of each word and affix is ​​used as the root node, the remaining characters are used as child nodes, the root node is used as the key of the hash table, and the child nodes are used as the value of the hash table to construct a hash tree. Figure 5 This is a schematic diagram of the structure of the hash tree provided in the embodiment of this application. Figure 5 , The multiple pre - preset words to be detected include: "in advance", "pre - repayment", "beware of being deceived", "warm reminder". The multiple affixes to be detected include "settle", "all", "select", etc. The first character of each of "in advance", "pre - repayment", "beware of being deceived", "warm reminder", "settle", "all", and "select" serves as the root node of the hash tree, namely "提", "温", "结", "全", "选". The other characters except the first character serve as child nodes. In the hash tree, the child node corresponding to the last character of a word or affix includes the vocabulary composed of all the characters in the link where the child node is located, as well as the word type. The word type is used to characterize whether the vocabulary stored in the child node is a word or an affix. For example, in the following structure, "提" serves as the root node, and its child node is "前". The child node "前" includes the vocabulary "提前". The word type of the vocabulary "提前" is neither a prefix nor a suffix. Therefore, the vocabulary "提前" is a word to be detected.

[0086]

[0087] In the embodiment of the present application, based on each character in the first text, traversing the tree - like structure can be achieved in the following way: First, read the i - th character from the first text. When the root node corresponding to the i - th character is traversed in the tree - like structure, determine the root node corresponding to the i - th character as the first node, where i is a positive integer increasing in sequence, 1 ≤ i ≤ I, and I is the number of characters in the first text; then, for the first node, read the (i + j) - th character from the first text. When there is a node corresponding to the (i + j) - th character among the child nodes of the first node, determine the node corresponding to the (i + j) - th character as the new first node, where j is a positive integer increasing in sequence, 1 ≤ j ≤ I - i; finally, for the first node, when there is no child node corresponding to the (i + j) - th character among the child nodes of the first node, determine the characters from the i - th character to the (i + j - 1) - th character as the continuously traversed characters.

[0088] Among them, I is a positive integer. Exemplarily, the hash tree includes multiple words to be detected: "in advance", "pre - repayment", "beware of being deceived", "warm reminder", and multiple affixes to be detected: "settle", "all", "select", etc. The first text is "提前结清是指在资源借用的期限内,用户选择提前偿还部分或全部余额的行为,银行允许提前结清,但是需要在1年后才可以”. First, read the first character "提" from the first text. Figure 5 Taking the hash tree as an example, the hash tree has a root node corresponding to the character "提". The root node corresponding to the first character "提" is determined as the first node. Read the second character "前" from the first text. The first node "提" has two child nodes "前" and "防". That is, there is a node corresponding to the second character "前" among the child nodes of the first node. The node corresponding to the second character "前" is determined as the new first node. Read the third character "结" from the first text. At this time, the first node "前" has a child node "偿". There is no child node corresponding to the third character "结" among the child nodes of the first node. The characters from the first character "提" to the second character "前" are determined as the continuously traversed characters.

[0089] In the embodiment of the present application, after determining the characters from the i-th character to the i + j - 1-th character as the continuously traversed characters, starting from the (i + j)-th character in the first text, continue to traverse in the tree structure until reaching the root node corresponding to a certain character, and repeat the above steps of obtaining the continuously traversed characters.

[0090] Exemplarily, read the third character "结" from the first text. The hash tree has a root node corresponding to the character "结". The root node corresponding to the third character "结" is determined as the first node. Read the fourth character "清" from the first text. The first node "结" has a child node "清". That is, there is a node corresponding to the fourth character "清" among the child nodes of the first node. The node corresponding to the fourth character "清" is determined as the new first node. Read the fifth character "是" from the first text. At this time, the first node "清" has no child nodes. There is no child node corresponding to the fifth character "是" among the child nodes of the first node. The characters from the third character "结" to the fourth character "清" are determined as the continuously traversed characters.

[0091] In step 1012, the characters continuously traversed from the tree structure are determined as the first word or the first affix.

[0092] In the embodiment of the present application, determining the characters continuously traversed from the tree structure as the first word or the first affix can be achieved through the following method: First, form the continuously traversed characters into a second word, and determine the last character in the second word; then, obtain the word type set for the last character from the tree structure, and based on the word type, determine the second word as the first word or the first affix.

[0093] Here, in the tree structure, the child node corresponding to the last character of a word or an affix includes the vocabulary composed of all the characters in the link where the child node is located, as well as the word type. The link includes all the nodes from the root node to this child node. The word type is used to characterize whether the vocabulary stored in this child node is a word or an affix. For example, Figure 5 In the hash tree, "提" serves as the root node, and its child node is "前". The child node "前" includes the vocabulary "提前", and the word type of the vocabulary "提前" is a word. Therefore, the vocabulary "提前" is the word to be detected.

[0094] Exemplarily, it is determined that the continuously traversed characters are the 1st character "提" to the 2nd character "前", and the continuously traversed characters are combined into a second word "提前". The last character in the second word is "前". The child node corresponding to the last character "前" is obtained from the tree structure, and the word type stored in this child node is a word. Then, the second word "提前" is determined as the first word in the first text. Alternatively, it is determined that the continuously traversed characters are the 3rd character "结" to the 4th character "清", and the continuously traversed characters are combined into a second word "结清". The last character in the second word is "清". The child node corresponding to the last character "清" is obtained from the tree structure, and the word type stored in this child node is a suffix. A suffix is a type of affix. Therefore, the second word "结清" is determined as the first affix in the first text.

[0095] By traversing the tree structure character by character in the embodiments of this application, the first word and the first affix in the first text can be accurately matched, avoiding misjudgment caused by character skipping or omission, and significantly improving the accuracy of detection.

[0096] In some embodiments, to determine the first word and the first affix in the first text in step 101, it can also be achieved through the following method: First, identify the language type of the first text, and determine the corresponding word library. The word library includes the words and affixes to be detected. Then, query the word library based on the words in the first text. Finally, when a word to be detected that is the same as the word is found in the query, the word is determined as the first word; when an affix to be detected that is the same as the word is found in the query, the word is determined as the first affix.

[0097] Here, the language type refers to the natural language used in the first text. Different natural languages ​​have different grammar, vocabulary, and writing rules. Language types include, but are not limited to, Chinese, English, and French. The language type of the first text can be determined using a language recognition algorithm or tool. For example, the language type of the first text can be determined using the language recognition function in a natural language processing library. Select a vocabulary corresponding to the language type of the first text. A vocabulary refers to a set of preset words to be tested and affixes to be tested. For example, if the language type is Chinese, select a Chinese vocabulary. Perform word segmentation on the first text, breaking it into at least one word. Word segmentation can be performed using a word segmentation tool. Compare each segmented word with the words to be tested and the affixes to be tested in the vocabulary. For each segmented word, if there is a word to be tested identical to the word in the vocabulary, the word is determined to be the first word; if an affix to be tested identical to the word is found in the query, the word is determined to be the first affix.

[0098] The embodiment of the present application can more accurately identify the first word and the first affix in the first text through word segmentation processing and vocabulary query, avoid misjudgment due to incorrect affixes or improper word combinations, significantly improve the accuracy of detection, and adapt to texts of different language types, thereby enhancing the robustness and scope of applicability of the detection method.

[0099] In step 102 , a first index interval of a first word and a second index interval of a first affix are determined based on first indexes of characters in the first text.

[0100] Here, for each character, the first index of the character in the first text is used to indicate the position of the character in the first text. Starting from the first character in the first text, a first index is assigned to each character, and the first index starts from 0 and increases sequentially. A character index table can be created to record the first index of each character in the first text. For each first word, the first index of the first character and the first index of the last character in the first word are obtained, and the first index of the first character is used as the left endpoint value and the first index of the last character is used as the right endpoint value to obtain the first index interval of the first word. That is, the first index interval of the first word is [the first index of the first character in the first word, the first index of the last character]. The process of determining the second index interval of the first affix is ​​the same as the process of determining the first index interval of the first word.

[0101] For example, for the first text "Early settlement refers to the user's decision to repay part or all of the balance in advance during the borrowed resource period. The bank allows early settlement, but only after one year," the first indexes for early settlement are 0, 1, 2, and 3, respectively. Therefore, the first index interval for the first word "early" is [0, 1], and the second index interval for the first affix "settlement" is [2, 3].

[0102] In step 103, based on the first index interval and the second index interval, the positional relationship between the first word and the first affix is ​​determined.

[0103] In some embodiments, see Figure 6 , Figure 6 It is shown that in step 103, the positional relationship between the first word and the first affix is ​​determined based on the first index interval and the second index interval, which can be achieved by the following steps 1031 to 1033:

[0104] In step 1031 , a first value is determined based on a maximum endpoint value of a first index interval and a first set value.

[0105] Here, the first set value is 1. The maximum endpoint value of the first index interval is the right endpoint value of the first index interval, that is, the largest first index in the first index interval. The sum of the maximum endpoint value and the first set value is determined as the first value.

[0106] For example, the first index interval of the first word "early repayment" is [20, 23], the maximum endpoint value is 23, and the first value is the sum of the maximum endpoint value and the first set value 23+1=24.

[0107] In step 1032 , a second value is determined based on the minimum endpoint value of the first index interval and the first set value.

[0108] Here, the minimum endpoint value of the first index interval is the left endpoint value of the first index interval, that is, the minimum first index in the first index interval. The difference between the minimum endpoint value and the first set value is determined as the second value.

[0109] For example, the first index interval of the first word "early repayment" is [20, 23], the minimum endpoint value is 20, and the second value is the difference between the minimum endpoint value and the first set value 20-1=19.

[0110] In step 1033, when the first numerical value is the same as the endpoint value of the second index interval, or the second numerical value is the same as the endpoint value of the second index interval, a positional relationship representing the proximity of the first word and the first affix is ​​determined.

[0111] Here, for each first word, determine whether the first numerical value or the second numerical value corresponding to the first word is the same as the endpoint value of the second index interval of each first affix in the first text. The endpoint value of the second index interval includes a left endpoint value and a right endpoint value. If there is a first affix in the first text whose right endpoint value of the second index interval is the same as the second numerical value corresponding to the first word, then the first affix is ​​adjacent to the first word, and the first affix is ​​the first prefix adjacent to the first word. If there is a first affix in the first text whose left endpoint value of the second index interval is the same as the first numerical value corresponding to the first word, then the first affix is ​​adjacent to the first word, and the first affix is ​​the first suffix adjacent to the first word.

[0112] If the endpoint values ​​of each first affix in the first text are different from the first numerical value and the second numerical value corresponding to the first word, then the first word has no adjacent affixes in the first text. At this time, a verification result is obtained that characterizes the first word as an abnormal word, and then a detection result of the first text anomaly is obtained.

[0113] For example, the first index interval for the first word "early repayment" is [20, 23], the first value is 24, and the second value is 19. If the second index interval for the first affix "select" in the first text is [18, 19], the right endpoint value is 21, which is the same as the second value. The first affix "select" and the first word "early repayment" are adjacent, and the first affix "select" is the first prefix adjacent to the first word "early repayment." If the second index interval for the first affix "all" in the first text is [24, 25], the left endpoint value is 24, which is the same as the first value. The first affix "all" and the first word "early repayment" are adjacent, and the first affix "all" is the first suffix adjacent to the first word "early repayment."

[0114] The embodiment of the present application accurately calculates the index intervals of the first word and the first affix, and determines the positional relationship between the first word and the first affix based on these index intervals. It can more accurately identify whether the first word and the first affix are adjacent, avoids text anomaly misjudgment due to ambiguous positional relationship, and significantly improves the accuracy of text detection.

[0115] In step 104 , when the positional relationship indicates that the first word and the first affix are adjacent, the first affix is ​​verified based on the correct affix set for the first word to obtain a detection result of the first text.

[0116] Here, based on the correct affix set for the first word, the first affix is ​​verified to obtain the detection result of the first text, which can be achieved in the following way: verify whether the first affix is ​​the same as the correct affix set for the first word. If the first affix is ​​the same as the correct affix set for the first word, a normal verification result for the first word is obtained; if the first affix is ​​different from the correct affix set for the first word, an abnormal verification result for the first word is obtained. After obtaining the verification result of each first word in the first text, the detection result of the first text is determined based on the verification result of each first word. Specifically, if the verification result of each first word in the first text is normal, the normal detection result of the first text is determined; if the verification result of any first word in the first text is abnormal, the abnormal detection result of the first text is determined.

[0117] The embodiment of the present application can more accurately identify the association between words and affixes by determining the first index interval of the first word and the second index interval of the first affix in the first text, and analyzing the positional relationship between the first word and the first affix based on the first index interval and the second index interval. When the first word is adjacent to the first affix, verification is performed based on a preset correct affix standard, thereby obtaining the detection result of the first text by detecting whether the affix adjacent to the first word is wrong. Compared with the related art that only detects whether there are keywords in the text to determine whether the text is abnormal, the present application adds affix detection, thereby improving the accuracy and reliability of text detection.

[0118] In some embodiments, the first affix includes a first prefix and a first suffix, and the correct affix includes a second prefix and a second suffix. In step 104, based on the correct affix set for the first word, the first affix is ​​verified to obtain a detection result of the first text, which can also be achieved in the following manner: first, based on the second prefix, the first prefix is ​​verified, and based on the second suffix, the first suffix is ​​verified; then, when the second prefix is ​​the same as the first prefix, and the second suffix is ​​the same as the first suffix, a detection result representing that the first text is normal is determined; when the second prefix is ​​different from the first prefix, or the second suffix is ​​different from the first suffix, a detection result representing that the first text is abnormal is determined.

[0119] Here, the first affix includes a first prefix and a first suffix. The first prefix is ​​a preset affix that needs to be located in front of the word to be detected, and the first suffix is ​​a preset affix that needs to be located after the word to be detected. The second prefix and second suffix included in the correct affix are similar. For example, for the word to be detected "early settlement", the correct second prefix is ​​"select", and the correct second suffix is ​​"all". For each first word in the first text, if it is determined that the first prefix and the first suffix are adjacent to the positional relationship of the first word, then based on the second prefix, the first prefix is ​​verified, which can be achieved in the following way: verify whether the second prefix is ​​the same as the first prefix; based on the second suffix, the first suffix is ​​verified, which can be achieved in the following way: verify whether the second suffix is ​​the same as the first suffix. When the second prefix is ​​the same as the first prefix, and the second suffix is ​​the same as the first suffix, the verification result is that the first word is normal. If the verification results of each first word in the first text are normal, the detection result representing that the first text is normal is determined; or, when the second prefix is ​​different from the first prefix, or the second suffix is ​​different from the first suffix, the verification result is that the first word is abnormal, and the detection result representing that the first text is abnormal is determined.

[0120] For example, the first text is "The bank allows early settlement, but only after one year." For the first word "in advance" in the first text, the adjacent first prefix is ​​"allow," and the first suffix is ​​"clear." Assuming the default correct second prefix is ​​"select," and the default correct second suffix is ​​"clear," then for the first word "in advance" in the first text, the first prefix and second prefix are different. Even if the first and second suffixes are the same, the verification result for the first word "in advance" is still abnormal, and the detection result indicating the first text is abnormal is determined.

[0121] The embodiment of the present application can more accurately identify the correctness of the first affix adjacent to the first word by separately checking the first prefix and the first suffix, avoids missing abnormal situations caused by affix errors, and significantly improves the accuracy of text detection.

[0122] In some embodiments, Figure 7 This is a flowchart of the text detection method provided in the embodiment of the present application. Figure 4 See also Figure 7 Before determining the first word and the first affix in the first text in step 101, the method includes the following steps 201 to 204:

[0123] In step 201, the web page document is converted into a node tree.

[0124] The node tree includes a third node representing text and a fourth node representing a tag. The web page document includes text and tags.

[0125] Here, the web page document is a document written in HTML language and displayed in the browser, and is also a document that requires text detection. Web page documents contain a variety of elements such as text, images, links, etc., which are used to display information and interactive functions. You can use an HTML parser (such as the Jsoup tool) to parse the web page document into a Document Object Model (DOM) tree structure, which is a node tree. The node tree is a data structure used to represent the hierarchical relationship of elements in a web page document. Each node in the node tree can be a third node representing text or a fourth node representing a tag, and the relationship between the nodes constitutes a tree structure. Text refers to the readable text content in a web page document, and the third node representing text contains the actual text content for displaying information. Tags are the basic elements in web page documents and are used to define the structure and content of web page documents. Tags are usually surrounded by angle brackets < and > and can contain attributes and content. Tags can be nested to form a hierarchical structure. For example : paragraph tag, : Block-level container tag. : Row container tag.

[0126] Figure 8 This is a schematic diagram of a node tree provided in an embodiment of the present application. Figure 8 The number of nodes includes the root node, multiple third nodes (TextNode), multiple fourth nodes (Node(p)), as well as block-level container nodes (Node(div)), table nodes (Node(table)), row container nodes (Node(span)), and some other nodes. Block-level container nodes (Node(div)), table nodes (Node(table)), row container nodes (Node(span)), and other nodes can all be considered segment nodes. That is, when a segment node is encountered, the text content before and after the segment node is split into two paragraphs.

[0127] In step 202 , for each third node, if the ancestor node of the third node includes any fourth node, the attribute value of the any fourth node is determined as the attribute value of the third node.

[0128] Here, after obtaining the node tree, a post-order traversal can be used to traverse the node tree to obtain multiple third nodes in the node tree. Ancestor nodes refer to all nodes in the node tree that are traced upward from a particular node. Ancestor nodes include parent nodes, grandparent nodes, and so on. The ancestor nodes of a third node are all nodes in the node tree that are traced upward from the third node. Attribute values ​​are numerical values ​​used to classify nodes in the node tree. Text in third nodes with the same attribute value belongs to the same paragraph.

[0129] For each third node, obtain at least one ancestor node of the third node from the node tree. If there is a fourth node in at least one ancestor node, the attribute value of the fourth node is determined as the attribute value of the third node. It should be noted that the attribute value of each fourth node in the node tree is different, and the specific value of the attribute value can be set by yourself. For example, see Figure 8 For the third node g1-1, the grandparent node of the third node g1-1 is the fourth node. Assuming that the attribute value of the fourth node is "123", the attribute value of the third node g1-1 is 123.

[0130] In step 203 , for each third node, if the ancestor node of the third node does not include the fourth node, the third numerical value is determined as the attribute value of the third node.

[0131] The third value is a value different from the attribute value of the fourth node.

[0132] Here, for each third node, at least one ancestor node of the third node is obtained from the node tree, and if the fourth node does not exist in at least one ancestor node, a preset third numerical value is obtained. The third numerical value is determined as the attribute value of the third node. When traversing to the fourth node, the preset fourth numerical value is determined as the new third numerical value, and when the fourth node does not exist in the ancestor node of the next third node, the new third numerical value is determined as the attribute value of the third node. It should be noted that the embodiment of the present application does not limit the specific values ​​of the third numerical value and the fourth numerical value, and can be set by itself, as long as the third numerical value is different from the attribute value of each fourth node in the node tree, the fourth numerical value is different from the attribute value of each fourth node in the node tree, and the fourth numerical value is different from the third numerical value.

[0133] For example, see Figure 8 , for the third node g2-1, the parent node of the third node g2-1 is a block-level container node, the grandparent node is the root node, and there is no fourth node in the ancestor node of the third node g2-1. At this time, the preset third value obtained is "abc", that is, the attribute value of the third node g2-1 is abc. Similarly, the attribute values ​​of the third nodes g2-2 to g2-4 are all abc. At this time, traverse to the fourth node, obtain the preset fourth value "efg", and use the fourth value "efg" as the new third value. For the traversed third node g4-1, there is no fourth node in the ancestor node of the third node g4-1. At this time, the third value obtained is "efg", that is, the attribute value of the third node g4-1 is efg.

[0134] In step 204 , the third nodes are clustered based on the attribute values ​​of the third nodes to obtain clusters, and the texts corresponding to the third nodes included in the clusters are combined into first texts.

[0135] Here, a cluster is a set of third nodes that contain at least one identical attribute value. After determining the attribute value of each third node in the node tree, the third nodes are clustered to obtain clusters. This can be done by: dividing at least one third node with the same attribute value into a cluster. For each cluster, the text corresponding to at least one third node included in the cluster is sequentially combined to obtain the paragraph text corresponding to the cluster. The first text is any one paragraph text among the multiple paragraph texts.

[0136] The embodiment of the present application converts the web page document into a node tree and clusters the nodes based on their attribute values, thereby being able to more accurately extract and combine text content and avoid text extraction errors caused by nested tags and different attribute values.

[0137] In some embodiments, the first text includes multiple second texts, and the second text is the text corresponding to the third node representing the text in the web page document. Figure 9 It is a flowchart of the text detection method provided by an embodiment of this application. Figure 5 Refer to Figure 9 After verifying the first affix according to the correct affix set for the first word in step 104 and obtaining the detection result of the first text, the method includes the following steps 105 to step 106:

[0138] In step 105, when the detection result indicates that the first word is an abnormal word, determine the second index of the character in the first text in the second text, and use the first index interval of the first word as the third index interval of the abnormal word.

[0139] Here, for each first word, when the detection result indicates that the first word is an abnormal word, obtain the second index of each character in the first text in the second text, and use the first index interval of the first word as the third index interval of the abnormal word. Exemplarily, the first index interval of the first word "early repayment" is [20, 23]. If the first word "early repayment" is an abnormal word, then use the first index interval [20, 23] as the third index interval of the abnormal word.

[0140] For each character in the first text, the second index is used to indicate the position of the character in the second text. Exemplarily, Figure 10 It is an index representation diagram of the first index and the second index of the characters in the first text provided by an embodiment of this application. Refer to Figure 10 For the first text "Early settlement means that within the borrowing period of resources, the user chooses to repay part or all of the balance in advance", the first text includes four second texts: "Early", "Settlement means", "Within the borrowing period of resources, the user chooses", "To repay part or all of the balance in advance". For the first second text "Early", the second index of the character "提" in the second text is 0, and the second index of the character "前" in the second text is 1; for the second second text "Settlement means", the second index of the character "结" in the second text is 0.

[0141] In step 106, based on the second index and the third index interval, mark the abnormal word in the web document.

[0142] Here, based on the second index of each character in the abnormal word and the third index interval of the abnormal word, determine the third node in the web document that contains some or all of the abnormal words from multiple third nodes, and mark the abnormal word in the third node that contains some or all of the abnormal words. The marking method for marking the abnormal word in the embodiment of this application is not limited. For example, it can be underlined, bolded, highlighted, etc.

[0143] By determining the third index range of the abnormal words, the embodiments of the present application can more accurately locate and mark the abnormal words, avoid mislabeling caused by inaccurate indexing, and significantly improve the accuracy of anomaly detection. Marking the abnormal words in the web document by using the third index range and the second index can visually show the user which words have problems, help the user quickly locate and correct errors, and improve the user experience.

[0144] In some embodiments, referring to Figure 11 , in step 106, marking the abnormal words in the web document based on the second index and the third index range can be implemented through the following steps 1061 to 1064:

[0145] In step 1061, based on the third index range, the third text where the abnormal word is located is filtered out from multiple second texts.

[0146] Here, the third text where the abnormal word is located refers to the second text where some or all of the abnormal words exist, or the second text where at least one character of the abnormal word exists. The fourth index range of each second text can be determined based on the first index of the character. For each second text, based on the third index range of the abnormal word and the fourth index range of this second text, it is judged whether this second text is the third text. If there is an intersection between the third index range of the abnormal word and the fourth index range of this second text, the second text is determined as the third text.

[0147] Exemplarily, the third index range of the abnormal word "prepay" is [20, 23], and the fourth index range of the second text "the act of prepaying part or all of the balance" is [20, 33]. Since there is an intersection between the third index range [20, 23] and the fourth index range [20, 33], the second text "the act of prepaying part or all of the balance" is the third text.

[0148] In step 1062, based on the second index of the character in the second text in the first text, the third index of the character in the second text in the abnormal word is determined.

[0149] Here, from the second indexes of each character in the first text in the second text, the second indexes of each character in the abnormal word in the second text are filtered out as the third index. Exemplarily, assume that the first word "prepay" is the abnormal word. Among them, the second index of the character "pre" in the second text "prepay" is 0, and the second index of the character "pay" in the second text "prepay" is 1. Then the third index of the character "pre" in the abnormal word "prepay" is 0, and the third index of the character "pay" is 1.

[0150] In step 1063, based on the third index, the first character belonging to the abnormal word in the third text is determined.

[0151] Here, for each third text, the character corresponding to each third index in the third text is determined as the first character of the abnormal word in the third text.

[0152] Exemplarily, for the third text "the act of prepaying part or all of the balance", the third indexes are 20, 21, 22, and 23, and the multiple first characters corresponding to the multiple third indexes are determined as "提", "前", "偿", "还".

[0153] In step 1064, the first character is marked in the web document.

[0154] Here, the first character can be marked in the web document by adding a label to the third text.

[0155] In the embodiment of the present application, marking the first character in the web document can be implemented by the following method: for each third text, perform the following processing: First, add a label to the position immediately before and immediately after the first character in the third text, and split the third text based on the label to obtain a fourth text and a fifth text, where the fourth text is the text composed of the label and the first character, and the fifth text is the text in the third text that is different from the fourth text; then, convert the fourth text into a fifth node and convert the fifth text into a sixth node; finally, replace the node corresponding to the third text in the web document with the fifth node and the sixth node, and mark the first character in the web document based on the label in the fifth node.

[0156] Here, for each third text, add a label to the position immediately before the first character among the multiple first characters, and add a label to the position immediately after the last character among the multiple first characters, to obtain the third text after adding the label. The label is an element used to mark the text enclosed by the label in the web document. Split the third text after adding the label to obtain a fourth text composed of the label and the first character, and a fifth text that is the remaining part of the third text except the fourth text. The fourth text can be converted into a fifth node, and the fifth text can be converted into a sixth node based on an HTML parser (such as the Jsoup tool). Replace the node corresponding to the third text in the node tree corresponding to the web document with the fifth node and the sixth node to obtain a new node tree. Convert the new node tree back into a new web document based on an HTML parser (such as the Jsoup tool), display the web document in a browser, and the first character in the displayed web document will display the mark corresponding to the label, such as bold or highlighted, etc.

[0157] Exemplarily, the label is <standard-mark>, the third text is "the act of prepaying part or all of the balance". If the first character among multiple first characters is "提" and the last character is "还", then the third text after adding the label is " <standard-mark>Early repayment <standard-mark>The behavior of part or all of the balance". The fourth text obtained after separation is" <standard-mark>Early repayment <standard-mark>”, and the fifth text is “the act of paying off part or all of the balance”. After completing the above steps, “early repayment” is highlighted in the web document.

[0158] The embodiment of the present application adds labels to the previous position and the next position adjacent to the first character in the third text, and segments the third text based on the labels to obtain the fourth text and the fifth text, then converts the fourth text into the fifth node, and converts the fifth text into the sixth node. Finally, the nodes corresponding to the third text in the web document are replaced with the fifth node and the sixth node, and the first character is marked in the web document based on the label in the fifth node. This can significantly improve the positioning accuracy and marking efficiency of abnormal characters, enhance the user's visual recognition of abnormal characters, and improve the user experience and the reliability of text processing.

[0159] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0160] Whether the knowledge entry is correct or not, and whether it complies with the specifications, is particularly important for knowledge managers and knowledge users. If the knowledge user obtains incorrect knowledge, it will directly affect the user's decision-making, and the trust of the knowledge manager in the team will be greatly reduced. The knowledge center is a platform that provides knowledge maintenance and management. In addition to providing knowledge management capabilities to system users, the knowledge center is also used to detect non-standard areas in the knowledge content and prompt it to the user. To this end, the embodiment of the present application provides a text detection method, that is, a knowledge specification detection method, which is applied to the knowledge center and can detect areas in the knowledge (corresponding to the web page document in the above embodiment) that do not comply with the specifications.

[0161] In a web page, the style of a paragraph content will be modified by multiple HTML tags. For example, the HTML text of "Today's weather (the weather is displayed in red, other text is displayed in black) is really good (it is really good and displayed in enlarged font size)" is: Today <span style＝"颜色：红色；”>weather <span style＝"字号为18；”>very nice Ah. Today, weather, really good, ah, are in different HTML tags respectively. If the agreement requires that "weather" must be followed by "really good" and must be preceded by "today", then when checking the content standardization of this text, you need to remove all HTML tags to know whether "weather" is followed by "really good" and preceded by "today"; however, if it is found that "weather" is not followed by "really good" (but "really bad"), then you need to tag "weather", mark the word "weather" as non-standard, and then display it to inform the user. But suppose the HTML text is: Today <span style＝"颜色：红色；”>sky The air is really nice Ah. Since the characters "tian" and "qi" in the word "weather" are distributed in different HTML tags, if "weather" does not meet the specification and needs to be marked with an irregular tag, it is necessary to separately mark the irregular tags in the HTML tags where "tian" and "qi" are located. In order to implement the specification detection of knowledge content when the content crosses HTML tags, an embodiment of the present application proposes a text detection method.

[0162] The text detection method of the embodiment of the present application is introduced below. The text detection method can be implemented in the following manner: First, text paragraph extraction. Based on the Jsoup tool, convert the HTML text (corresponding to the web page document in the above embodiment) into a Document object (corresponding to the node tree in the above embodiment), then perform segmentation processing on the HTML text, and sequentially obtain the text nodes of each segment (corresponding to the third node in the above embodiment) to generate the paragraph text content (corresponding to the first text in the above embodiment). At the same time, since each paragraph is composed of one or more text nodes, all the text nodes in the paragraph are parsed. Then, paragraph character index table. Construct a paragraph character index (corresponding to the first index in the above embodiment) table, a text node character index (corresponding to the second index in the above embodiment) table, and a mapping table between paragraph characters and text node characters. Then, the specification detection word hash tree: load the specification detection words (corresponding to the words to be detected in the above embodiment), construct the specification detection word hash tree, build all the specification detection words into a tree, the root node is a virtual node, the secondary nodes (the children of the root node) are stored in the hash table, and its subtree is used as the value of the hash table. The node where the last character of each specification detection word is located stores the specification detection word. Secondly, perform specification detection: determine whether the paragraph meets the content specification. Finally, return the detection result: convert the Document object back into HTML text.

[0163] Figure 12 It is a schematic diagram of the implementation of the text detection method provided by the embodiment of the present application.

[0164] Step 301, rich text content.

[0165] Among them, obtain the knowledge content to be subjected to specification detection, that is, the rich text content (corresponding to the web page document in the above embodiment).

[0166] Step  302, text paragraph extraction.

[0167] Among them, for the specification detection of the knowledge content, it is necessary to first use the Jsoup tool to extract the paragraph content from the HTML text. See Figure 8 Based on the Jsoup tool, the HTML text is converted into a Document object to obtain an HTML document node tree (corresponding to the node tree in the above embodiment), where TextNode is a text node, Node(p) is a paragraph tag node, and the others can be uniformly regarded as segment nodes (that is, when encountering these nodes, the previous and next content will be split into two segments). Using the tree post-order traversal method, all p paragraph tag nodes (corresponding to the fourth node in the above embodiment) are set with the attribute value = "xxx", and all text nodes (corresponding to the third node in the above embodiment) are traversed and placed into the text node list. Figure 13 This is a schematic diagram of a text node list provided in an embodiment of the present application. Figure 13 In the process of traversing the text nodes, each text node is numbered to obtain text node g1-1, text node g1-2, text node g1-3, text node g2-1, text node g2-2, text node g2-3, text node g2-4, text node g3-1, text node g3-2, text node g3-3, text node g4-1, text node g5-1 and text node g6-1.

[0168] Figure 14 It is a schematic diagram of assigning attribute values ​​to text nodes provided by the present application embodiment. Create a global variable group (corresponding to the third numerical value in the above-described embodiment), traverse the text node list in sequence, and judge whether the parent node or ancestor node of the text node is a p paragraph tag node. If so, the attribute value of the p paragraph tag node is copied to the text node TextNode, and judge whether the global variable group is empty, and clear it if it is not empty; If not, obtain the value of the global variable group, and set it to the attribute value of the text node (if the global variable group is empty at this time, it is regenerated). Traverse all text nodes, set attribute values ​​for each text node. Exemplary, the attribute value 123 of the tag node is copied to text nodes g1-1, g1-2, and g1-3, the global variable abc is copied to text nodes g2-1, g2-2, g2-3, and g2-4, the attribute value 456 of the tag node is copied to text nodes g3-1, g3-2, and g3-3, and the global variable efg is copied to text node g4-1. The text nodes are grouped according to their attribute values, and each group of text nodes is regarded as a paragraph (corresponding to the clustering in the above embodiment). Then, the plain text in each text node is obtained and spliced ​​to obtain the completed paragraph plain text (corresponding to the first text in the above embodiment), and the paragraph text extraction is completed.

[0169] Step 303: paragraph character index table.

[0170] Among them, see Figure 10 , the headers of the paragraph character index table are paragraph characters (corresponding to the characters in the above embodiment), paragraph indexes (corresponding to the first index in the above embodiment) and fragment indexes (corresponding to the second index in the above embodiment). The paragraph characters represent each character in the paragraph text (or text node text), the paragraph index is the index position where the character appears in the paragraph (corresponding to the first text in the above embodiment), and the fragment index is the index position where the character appears in the text node (corresponding to the second text in the above embodiment). The arrows from the table to the text node set (multiple second texts) represent pointers from each index position to the text node. The effect that the entire paragraph character index table can achieve is that, through each character, the text node where each character appears and its position in the text node can be determined.

[0171] Step 304: construct a standard detection word hash tree.

[0172] Among them, multiple standard detection words and the prefix and suffix of each standard detection word are pre-set. All standard detection words and standard detection word prefixes and suffixes are loaded, and each character is used as a node to construct the following Figure 5 The hash tree shown in the figure uses the first character of each word as the root node, the root node as the key of the hash table, and the child nodes as the values ​​of the hash table. The child nodes of the same level are stored in a hash table to improve the query efficiency when there are many standard detection words. The hash tree can be represented by json format data, where the key in each hash structure is a character in the standard detection word, the vocabulary at the same level as the child node represents the words of the entire link, and the prefix and suffix represent whether the current word is a prefix or suffix.

[0173] Step 305: traverse the paragraph character index table.

[0174] Among them, traversing the paragraph character index table means traversing the entire paragraph text word by word. Every time a paragraph character is traversed, the hash table is checked to see if a tree can be obtained. If it can be obtained, the next character is traversed based on the currently obtained tree. Then, the hash table of the child node of the tree queried for the previous character is checked to see if this character exists. If it exists, the search continues until no child node is found. The example is as follows: For the text "Early settlement refers to the behavior of the user choosing to repay part or all of the balance in advance within the resource borrowing period", see Figure 15 , traverse the first character "提", and find the subtree rooted at "提" in the hash table (circled by a border in the figure). Continue traversing, and continue to search for the character "前" under the subtree rooted at "提". If it exists, continue traversing downwards. It is found that there is no character "结" among the subsequent child nodes, so the traversal ends. Then, query again from the top-level hash table to see if there is the character "结", see Figure 16 , find the subtree rooted at "结" in the hash table (circled by a border in the figure). Then traverse the character "清", and find the character "清" in the subtree rooted at "结", and obtain the canonical detection word suffix (corresponding to the first suffix in the above embodiment) "结清". Repeat the above matching steps, and finally obtain Figure 17 the canonical detection words (corresponding to the first word in the above embodiment), prefixes (corresponding to the first prefix in the above embodiment), and suffixes (corresponding to the first suffix in the above embodiment) as shown. The trees on the left side of the canonical detection word prefix and suffix are index values, which are used to indicate where the current word starts to appear in the paragraph.

[0175] Step 306, canonical detection.

[0176] Among them, start to perform canonical detection. First, obtain all the canonical detection words from the matching results. If there are no canonical detection words, it means that the entire paragraph text is canonical (corresponding to the detection result being the first text is normal in the above embodiment). If canonical detection words are obtained, then according to the index interval [begin, end] of the canonical detection words, go to the matching prefix and suffix results, obtain the suffix according to end + 1 (corresponding to the first numerical value in the above embodiment), obtain the prefix according to begin - 1 (corresponding to the second numerical value in the above embodiment), and end + 1 and begin - 1 are Figure 17 The numerical value on the left side of the prefix and suffix of the standard detection word. For example, advance [0,1], settle [2,3], repay in advance [20,23], all [24,25], select [18,19], where for the standard detection word "advance", the suffix "settle" is obtained according to end+1=1+1=2, and for the standard detection word "repay in advance", the suffix "all" is obtained according to end+1=23+1=24, and the prefix "select" is obtained according to begin-1=20-1=19. If the prefix and suffix of the standard detection word cannot be obtained, the standard detection word does not meet the standard (corresponding to the first word being an abnormal word in the above embodiment). If it can be obtained, the prefix and suffix of the standard detection word actually configured are queried from the database (corresponding to the correct affix in the above embodiment), and the standard detection word, prefix, and suffix are configured into a complete inspection rule in the database. Determine whether the obtained standard detection word prefix or suffix is ​​consistent with the actual configured standard detection word prefix and suffix. If they are consistent, they meet the standard, and if they are inconsistent, they do not meet the standard. For example, for early balance settlement, the standard detection word is "early balance settlement" and the suffix is ​​"borrowing." The text to be tested is: "xxx Bank allows early balance settlement, but only after one year of borrowing." The text contains "early balance settlement" and "borrowing," but "early balance settlement" is not followed by "borrowing." This is inconsistent with the actual standard detection word prefix and does not comply with the standard.

[0177] You can also mark the standard detection words that do not conform to the legal norms (corresponding to the abnormal words in the above embodiment). Mark the part of the plain text content of the text node relative to the index range interval by adding HTML tags. <standard-mark>< / standard-mark> The specific operation is as follows: According to the paragraph index interval of the standard detection word, go to the paragraph character index table to obtain the fragment index of each character. For example, the paragraph interval of "early settlement" is [0,4], and the corresponding text node index intervals are 0, 1, 0, 1. Therefore, the four characters "early settlement" span the tag node. According to the index pointer (i.e., the paragraph index), you can get which text node "early settlement" corresponds to. For example, the two characters "in advance" appear in the [0,1] index interval position of the corresponding text node, and the two characters "settlement" appear in the [0,1] index interval position of the corresponding text node. Call the TextNode.after(arg) method of the text node, and input an HTML fragment, and then call the TextNode.remove() method of the text node. That is, add the content of the standard detection tag after the text node. The Jsoup tool will convert the HTML text after the tag into an HTML node, and then remove the old text node, which is equivalent to replacing the text node with the content marked with the non-standard word.

[0178] Step 307: Return the detection result.

[0179] Because the above solution operates on nodes in the node tree, it ultimately directly calls the document object conversion method (Document.body().html() method) to convert the Document object into HTML text, marking any non-standard detection words. The terminal then applies special style processing to standard detection words, ensuring that the knowledge content is standardized according to the agreement and notifying the user of any non-standard words.

[0180] The embodiment of the present application can perform a standard check on the knowledge according to the rules before the knowledge is formally saved, thereby preventing non-standard knowledge from being put into production applications and avoiding the risks of using production knowledge.

[0181] The following continues to describe the exemplary structure of the text detection device 455 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the text detection device 455 of the memory 450 may include: a word determination module 4551, used to determine the first word and the first affix in the first text; an index determination module 4552, used to determine the first index interval of the first word and the second index interval of the first affix based on the first index of the character in the first text; a position determination module 4553, used to determine the positional relationship between the first word and the first affix based on the first index interval and the second index interval; a detection module 4554, used to verify the first affix based on the correct affix set for the first word when the positional relationship indicates that the first word and the first affix are adjacent, so as to obtain the detection result of the first text.

[0182] In some embodiments, the position determination module 4553 is also used to determine the first numerical value based on the maximum endpoint value of the first index interval and the first set value; determine the second numerical value based on the minimum endpoint value of the first index interval and the first set value; and determine the positional relationship representing the proximity of the first word and the first affix when the first numerical value is the same as the endpoint value of the second index interval, or the second numerical value is the same as the endpoint value of the second index interval.

[0183] In some embodiments, the first affix includes a first prefix and a first suffix, and the correct affix includes a second prefix and a second suffix. Detection module 4554 is further configured to verify the first prefix based on the second prefix, and to verify the first suffix based on the second suffix; if the second prefix is ​​the same as the first prefix, and the second suffix is ​​the same as the first suffix, determine a detection result indicating that the first text is normal; if the second prefix is ​​different from the first prefix, or the second suffix is ​​different from the first suffix, determine a detection result indicating that the first text is abnormal.

[0184] In some embodiments, the word determination module 4551 is further configured to traverse the tree structure based on each character in the first text; and determine the characters continuously traversed from the tree structure as the first word or the first affix.

[0185] In some embodiments, the word determination module 4551 is also used to read the i-th character from the first text, and when the root node corresponding to the i-th character is traversed from the tree structure, the root node corresponding to the i-th character is determined as the first node, wherein i is a positive integer that increases successively, 1≤i≤I, and I is the number of characters in the first text; for the first node, the i+j-th character is read from the first text, and when there is a node corresponding to the i+j-th character in the child nodes of the first node, the node corresponding to the i+j-th character is determined as the new first node, wherein j is a positive integer that increases successively, 1≤j≤Ii; for the first node, when there is no child node corresponding to the i+j-th character in the child nodes of the first node, the i-th character to the i+j-1-th character are determined as the characters that have been traversed continuously.

[0186] In some embodiments, the word determination module 4551 is also used to group the continuously traversed characters into a second word and determine the last character in the second word; obtain the word type set for the last character from the tree structure, and determine the second word as the first word or the first affix based on the word type.

[0187] In some embodiments, the text detection device 455 also includes a text conversion module for converting a web page document into a node tree, the node tree including a third node representing the text and a fourth node representing the tag, and the web page document includes text and the tag; for each third node, when the ancestor node of the third node includes any fourth node, the attribute value of any fourth node is determined as the attribute value of the third node; for each third node, when the ancestor node of the third node does not include the fourth node, the third numerical value is determined as the attribute value of the third node, wherein the third numerical value is a numerical value different from the attribute value of the fourth node; based on the attribute value of the third node, the third nodes are clustered to obtain clusters, and the texts corresponding to the third nodes included in the clusters are combined into the first text.

[0188] In some embodiments, the first text includes multiple second texts, and the second text is the text corresponding to the third node representing the text in the web document. The text detection device 455 also includes a marking module for determining the second index of a character in the first text in the second text when the detection result indicates that the first word is an abnormal word, and using the first index interval of the first word as the third index interval of the abnormal word; and marking the abnormal word in the web document based on the second index and the third index interval.

[0189] In some embodiments, the marking module is also used to filter out a third text containing abnormal words from multiple second texts based on a third index interval; determine the third index of the character in the abnormal word in the second text based on the second index of the character in the first text in the second text; determine the first character in the third text belonging to the abnormal word based on the third index; and mark the first character in the web page document.

[0190] In some embodiments, the marking module is also used to perform the following processing for each third text: adding labels to the previous position adjacent to the first character and the next position adjacent to the first character in the third text, and segmenting the third text based on the labels to obtain a fourth text and a fifth text, wherein the fourth text is a text composed of the label and the first character, and the fifth text is a text in the third text that is different from the fourth text; converting the fourth text into a fifth node, and converting the fifth text into a sixth node; replacing the node corresponding to the third text in the web page document with the fifth node and the sixth node, and marking the first character in the web page document based on the label in the fifth node.

[0191] In some embodiments, the word determination module 4551 is also used to identify the language type of the first text, determine the vocabulary corresponding to the language type, and the vocabulary includes words to be detected and affixes to be detected; query the vocabulary based on the words in the first text; when a word to be detected that is the same as the word is queried, the word is determined to be the first word; when affix to be detected that is the same as the word is queried, the word is determined to be the first affix.

[0192] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the text detection method described in the present invention.

[0193] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the text detection method provided by the embodiment of the present application, for example, Figure 3 The text detection method shown.

[0194] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0195] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0196] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0197] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0198] In summary, the embodiments of the present application can improve the accuracy and reliability of text detection, and can perform standardization checks on knowledge according to rules before the knowledge is formally saved, thereby avoiding the use of non-standard knowledge in production applications and circumventing the risks of using production knowledge.

[0199] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A text detection method, characterized in that: The method comprises: determining a first word and a first affix in a first text; Determining a first index interval of the first word and a second index interval of the first affix based on a first index of a character in the first text; Determining a positional relationship between the first word and the first affix based on the first index interval and the second index interval; When the positional relationship indicates that the first word is adjacent to the first affix, the first affix is ​​verified based on a correct affix set for the first word to obtain a detection result of the first text.

2. The method according to claim 1, characterized in that The determining, based on the first index interval and the second index interval, a positional relationship between the first word and the first affix includes: Determining a first value based on a maximum endpoint value of the first index interval and a first set value; Determining a second value based on the minimum endpoint value of the first index interval and the first set value; When the first numerical value is the same as the endpoint value of the second index interval, or the second numerical value is the same as the endpoint value of the second index interval, a positional relationship representing the proximity of the first word and the first affix is ​​determined.

3. The method according to claim 1, characterized in that The first affix includes a first prefix and a first suffix, and the correct affix includes a second prefix and a second suffix; The verifying the first affix based on the correct affix set for the first word to obtain a detection result of the first text includes: Verify the first prefix based on the second prefix, and verify the first suffix based on the second suffix; When the second prefix is ​​the same as the first prefix, and the second suffix is ​​the same as the first suffix, determining a detection result indicating that the first text is normal; In a case where the second prefix is ​​different from the first prefix, or the second suffix is ​​different from the first suffix, a detection result indicating that the first text is abnormal is determined.

4. The method according to claim 1, wherein Determining the first word and the first affix in the first text includes: traversing the tree structure based on each character in the first text; Characters continuously traversed from the tree structure are determined as the first word or the first affix.

5. The method according to claim 4, characterized in that The traversing the tree structure based on each character in the first text includes: Reading an i-th character from the first text, and when a root node corresponding to the i-th character is reached through traversing the tree structure, determining the root node corresponding to the i-th character as a first node, where i is a positive integer that increases in sequence, 1≤i≤I, and I is the number of characters in the first text; For the first node, read the i+jth character from the first text, and if a node corresponding to the i+jth character exists among the child nodes of the first node, determine the node corresponding to the i+jth character as a new first node, where j is a successively increasing positive integer, 1≤j≤Ii; For the first node, if there is no child node corresponding to the (i+j)th character among the child nodes of the first node, the (i)th character to the (i+j-1)th character are determined as the continuously traversed characters.

6. The method according to claim 4, characterized in that The step of determining the characters found in successive traversals of the tree structure as the first word or the first affix includes: Combining the continuously traversed characters into a second word, and determining the last character in the second word; A word type set for the last character is obtained from the tree structure, and based on the word type, the second word is determined to be the first word or the first affix.

7. The method according to any one of claims 1 to 6, characterized in that Before determining the first word and the first affix in the first text, the method further includes: Converting a web page document into a node tree, wherein the node tree includes a third node representing text and a fourth node representing a tag, wherein the web page document includes the text and the tag; For each third node, when the ancestor node of the third node includes any fourth node, determining the attribute value of the arbitrary fourth node as the attribute value of the third node; For each third node, if the ancestor node of the third node does not include the fourth node, determine a third value as the attribute value of the third node, wherein the third value is a value different from the attribute value of the fourth node; The third nodes are clustered based on the attribute values ​​of the third nodes to obtain clusters, and the texts corresponding to the third nodes included in the clusters are combined into the first texts.

8. The method according to any one of claims 1 to 6, characterized in that The first text includes a plurality of second texts, wherein the second text is a text corresponding to a third node representing a text in a web document; After verifying the first affix based on the correct affix set for the first word to obtain a detection result of the first text, the method further includes: If the detection result indicates that the first word is an abnormal word, determining a second index of a character in the first text in the second text, and using the first index interval of the first word as a third index interval of the abnormal word; Based on the second index and the third index interval, the abnormal words are marked in the web page document.

9. The method according to claim 8, characterized in that The marking of the abnormal words in the web document based on the second index and the third index interval includes: Based on the third index interval, filtering out the third text containing the abnormal word from the plurality of second texts; Determining the third index of the character in the abnormal word in the second text based on the second index of the character in the first text in the second text; Based on the third index, determining a first character in the third text that belongs to the abnormal word; The first character is marked in the web page document.

10. The method according to claim 9, characterized in that The marking of the first character in the webpage document includes: For each third text, perform the following processing: adding a label to a preceding position adjacent to the first character and a succeeding position adjacent to the first character in the third text, and segmenting the third text based on the labels to obtain a fourth text and a fifth text, wherein the fourth text is a text consisting of the label and the first character, and the fifth text is a text in the third text different from the fourth text; Convert the fourth text into a fifth node, and convert the fifth text into a sixth node; The node corresponding to the third text in the web document is replaced with the fifth node and the sixth node, and the first character is marked in the web document based on the label in the fifth node.

11. The method according to any one of claims 1 to 6, characterized in that: Determining the first word and the first affix in the first text includes: Identifying a language type of the first text and determining a vocabulary corresponding to the language type, wherein the vocabulary includes words and affixes to be detected; querying the vocabulary based on words in the first text; When the word to be detected that is the same as the word is found in the search, determining the word as the first word; When the affix to be detected that is the same as the word is found, the word is determined to be the first affix.

12. A text detection device, characterized in that: The device comprises: A word determination module, configured to determine a first word and a first affix in a first text; An index determination module, configured to determine a first index interval of the first word and a second index interval of the first affix based on a first index of a character in the first text; a position determination module, configured to determine a positional relationship between the first word and the first affix based on the first index interval and the second index interval; The detection module is configured to verify the first affix based on a correct affix set for the first word when the positional relationship indicates that the first word is adjacent to the first affix, so as to obtain a detection result of the first text.

13. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; The processor is configured to implement the text detection method according to any one of claims 1 to 11 when executing the computer-executable instructions or computer program stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the text detection method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the text detection method according to any one of claims 1 to 11 is implemented.

Images