Text retrieval methods, devices, electronic equipment and storage media
By constructing regular expressions containing search keywords and preset characters for text retrieval, the problem of text retrieval under the influence of sentence segmentation and special characters is solved, and higher recall and readability are achieved.
Patent Information
- Application Number
- CN202210784810.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-05
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-07-05
AI Technical Summary
Existing text retrieval methods cannot accurately recall text sentences containing search keywords, especially when affected by sentence breaks and special characters, resulting in poor retrieval results.
Regular expressions are used to construct search criteria, which include search keywords and their preset characters. These criteria are used to retrieve target strings that match the target text, avoiding the influence of sentence breaks and special characters.
It improves the recall rate of search keywords, ensures the accuracy and readability of search results, and can effectively recall text sentences containing search keywords in segmented or paragraphed text.
Smart Images

Figure CN116127101B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data retrieval technology, and in particular to a text retrieval method, apparatus, electronic device and storage medium. Background Technology
[0002] To facilitate semantic understanding of text, long texts are often segmented or have punctuation and special characters added during storage, such as text obtained through speech recognition, which contains numerous punctuation marks. Segmenting or adding punctuation to long texts also helps in retrieving the smallest single sentences containing only the target keywords. In some scenarios, it may be necessary to filter out a portion of the text, such as text containing certain keywords, and then highlight those keywords using special symbols. Therefore, how to retrieve desired text content from a text has become a hot research topic.
[0003] The conventional text retrieval method uses the content to be retrieved as search keywords, and then uses those keywords as search criteria to retrieve text content containing those keywords. However, this text retrieval scheme is not very effective and usually cannot accurately recall text sentences containing the search keywords. Summary of the Invention
[0004] Based on the aforementioned technological status, this application proposes a text retrieval method, apparatus, electronic device, and storage medium, aiming to reduce the impact of text segmentation and special characters on text retrieval and improve text retrieval results.
[0005] The first aspect of this application provides a text retrieval method, comprising: constructing a regular expression based on search keywords; wherein the regular expression is used to retrieve the search keywords and a preset string, the preset string being composed of characters included in the search keywords and preset characters; the preset characters in the preset string being located between any two adjacent characters in the search keywords; retrieving a target string from target text that matches the regular expression; the target text having the characteristic that adjacent characters belonging to the same word are separated by the preset characters; and determining a text sentence containing the retrieved target string from the target text.
[0006] A second aspect of this application provides a text retrieval device, comprising: a retrieval preparation unit, configured to construct a regular expression based on retrieval keywords; wherein the regular expression is used to retrieve the retrieval keywords and a preset string, the preset string being composed of characters included in the retrieval keywords and preset characters; in the preset string, the preset characters are located between any two adjacent characters in the retrieval keywords; a text retrieval unit, configured to retrieve a target string from target text that matches the regular expression; the target text having the characteristic that adjacent characters belonging to the same word are separated by the preset characters; and a retrieval processing unit, configured to determine a text sentence containing the target string from the target text.
[0007] A third aspect of this application provides an electronic device, comprising: a memory and a processor; wherein the memory is connected to the processor and is used to store a computer program; the processor is used to implement the above-described text retrieval method by running the computer program in the memory.
[0008] A fourth aspect of this application provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described text retrieval method.
[0009] The text retrieval method proposed in this application does not directly search the target text using the search keywords. Instead, it first constructs a regular expression based on the search keywords. This regular expression not only retrieves the search keywords from the target text but also retrieves a predefined string composed of characters included in the search keywords and predefined characters. Further, based on this regular expression, it retrieves target strings from the target text that match it. Finally, by identifying the text sentences containing the retrieved target strings from the target text, the retrieval results corresponding to the search keywords are obtained.
[0010] Because regular expressions offer greater tolerance and flexibility for string retrieval, this text retrieval method, when applied to target text where adjacent characters belonging to the same word are separated by preset characters, can not only retrieve the search keywords from the target text, but also, even if the search keywords are mixed with one or more preset characters, it can still retrieve strings containing both the search keywords and preset characters, thus improving the recall rate for the search keywords. Applying this retrieval method to text retrieval scenarios with sentence breaks or paragraph breaks can avoid the impact of punctuation marks, paragraph breaks, or sentence breaks on the retrieval of search keywords, thereby significantly improving text retrieval performance. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0012] Figure 1 A schematic diagram of an implementation environment for the text retrieval scheme provided in this application embodiment;
[0013] Figure 2 This is a schematic diagram illustrating an application scenario of the text retrieval scheme provided in the embodiments of this application;
[0014] Figure 3 This is a schematic diagram of a text retrieval application interface provided in an embodiment of this application;
[0015] Figure 4 A flowchart illustrating a text retrieval method provided in an embodiment of this application;
[0016] Figure 5 A flowchart illustrating another text retrieval method provided in this application embodiment;
[0017] Figure 6 A flowchart illustrating yet another text retrieval method provided in this application embodiment;
[0018] Figure 7 A schematic diagram illustrating the processing procedure of the text retrieval method provided in this application embodiment when applied to a real-world scenario;
[0019] Figure 8 This is a schematic diagram of the structure of a text retrieval device provided in an embodiment of this application;
[0020] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0021] The technical solutions of this application are applicable to text retrieval application scenarios. By adopting the technical solutions of this application, text sentences containing search keywords can be retrieved more comprehensively and accurately from the target text.
[0022] Generally speaking, text retrieval refers to finding text sentences or paragraphs containing text of interest from target text. A typical text retrieval strategy involves using the desired keywords as search criteria to search the target text, finding strings that perfectly match the search criteria, thus obtaining the search results.
[0023] However, in some cases, two adjacent characters that originally belong to the same word in the text may be separated by certain characters, which will directly result in the inability to retrieve the word from the text by using the word as a search condition.
[0024] For example, in the text "Today's sunshine is so warm, like the sunshine of spring. It's very rare to have such good weather in the dead of winter. Coincidentally, the company organized a mountain climbing activity today. After the climb started, my body warmed up quickly, and I felt full of energy," if the search keyword is the word "warmth," then when searching for the word "warmth" in this text, according to a conventional document retrieval scheme, the word "warmth" will be compared with each character in the above text segment in turn to determine whether the compared character is the word "warmth." If it is, the sentence containing the matched character will be recalled as the matched text sentence; otherwise, the comparison of subsequent characters will continue.
[0025] According to the above search scheme, using the word "warm" as a search condition, no search results can be retrieved from the above text. The fundamental reason is that the word "warm" in the above text is separated by "," and the current search scheme cannot identify the match between "warm" and "warm" in the text, so no search results can be obtained.
[0026] The above situation is most common in scenarios involving retrieval of text from speech recognition. Due to the accuracy limitations of speech recognition, punctuation marks or sentence breaks may be added at inappropriate locations in the text, causing adjacent characters that originally belonged to the same word to be separated by punctuation or divided into different sentences. Therefore, when attempting to retrieve this word from the speech text using conventional text retrieval methods, it will be impossible to find the word using this word as a search condition.
[0027] In other scenarios, such as sensitive information screening and voice / text quality inspection, users often evade screening by adding special symbols to inappropriate content or by deliberately speaking in a way that deviates from standard language usage to avoid voice quality inspection. These scenarios generate a large amount of text where adjacent characters that originally belong to the same word are separated, making effective text retrieval difficult with current search technologies.
[0028] To address the aforementioned text retrieval scenarios and ensure high recall rates, some researchers have proposed removing special characters (excluding regular characters) from text segments or omitting sentence breaks. This would avoid the impact of special characters or sentence breaks on text retrieval. However, text without special characters or sentence breaks often has poor readability, and the recalled text sentences are lengthy and difficult for users to understand.
[0029] For example, if all the punctuation marks in the above text are removed, the text becomes: "Today's sunshine is so warm, like the sunshine of spring. It's very rare to have such good weather in the dead of winter. Coincidentally, the company organized a mountain climbing activity today. After the climb started, my body warmed up quickly, and I felt full of energy." Clearly, this text is very difficult to read, and users need to punctuate and understand it themselves.
[0030] On the other hand, if the word "warmth" is searched for in this text segment, since the entire text segment is not punctuated, when the word "warmth" is found in the text segment, the entire text segment will be recalled. That is, the recalled text sentence is "Today's sunshine is so warm, like the sunshine in spring. It is very rare to have such good weather in the dead of winter. Coincidentally, the company organized a mountain climbing activity today. After the climb started, my body warmed up quickly and I felt full of energy." At this time, the user still needs to confirm where the word "warmth" is in the recalled text or in which sentence, which is very inconvenient.
[0031] It is evident that conventional text retrieval methods are highly susceptible to the effects of text segmentation and special characters, making it difficult to accurately retrieve text sentences containing search keywords. However, failing to segment the text or add special characters will affect the readability of the search results, failing to meet users' needs for intuitive and comprehensive retrieval of text sentences containing search keywords from the target text.
[0032] Based on the aforementioned technological status, this application proposes a novel text retrieval scheme. This scheme optimizes the retrieval of text containing special characters and sentence breaks. Specifically, it optimizes the search criteria, allowing them to include not only characters contained in the search keywords themselves but also other special characters. This makes the search criteria more generalized and tolerant. Then, the optimized search criteria are used to retrieve matching strings from the target text. This processing ensures that keyword retrieval is unaffected by special characters. Even if keywords are segmented by special characters in the target text, the segmented keywords can still be retrieved during the search. Therefore, this method can guarantee both the recall rate of the search results and the ability to recall single sentences containing the search keywords from long target texts, ensuring the readability of the search results.
[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0034] Exemplary Implementation Environment
[0035] Figure 1 An exemplary implementation environment of this application is shown, which is a data retrieval system. See also Figure 1 As shown, the system mainly includes one or more terminal devices 001, a server 002, and a database 003.
[0036] Terminal device 001 can include various types of computer equipment, such as smart terminal devices (e.g., portable handheld devices), computers (personal computers or laptops), self-service terminals, smart robots, vehicle-mounted terminals, and wearable devices. Specifically, smart terminal devices can include smartphones, tablets, personal digital assistants (PDAs), etc.; self-service terminals can include self-service inquiry terminals, self-service terminals, etc.; smart robots can include service robots, etc.; and wearable devices can include head-mounted displays, smartwatches, smart bracelets, and other devices.
[0037] Terminal device 001 can install and run applications that support text retrieval, such as system applications, various Internet-related applications, communication applications (e.g., email applications), and short message service (SMS) applications, and can use various communication protocols. For example, terminal device 001 can obtain search keywords input by the user, and then, through interaction with server 002 or through local data retrieval, terminal device 001 obtains search results for the search keywords and outputs the results. Throughout the process, terminal device 001 can engage in any form of human-computer interaction with the user, such as through data interaction methods like images, text, and voice.
[0038] Server 002 can run one or more services or applications capable of performing text retrieval. Alternatively, server 002 can be a distributed system server, cloud server, workstation, or an intelligent cloud computing server or intelligent cloud host based on artificial intelligence technology. The number of servers 002 can be set to one or more as needed.
[0039] Server 002 is used to provide background services for applications that support text retrieval. For example, it can provide text retrieval services to terminal device 001, or provide text data support, text retrieval algorithm support, etc.
[0040] Database 003 serves as the data center of the entire data retrieval system, primarily used for data storage, specifically storing various types of data such as audio data, text data, and image data. Database 003 can reside in various locations. For example, the database used by server 002 can be local to server 002, or it can be located remotely to server 002 and communicate with server 002 via a network-based or dedicated connection. Database 003 can be of any type and in any number of databases. In some embodiments, the databases used by server 002 can be relational databases. One or more of these databases can store, update, and retrieve data from and from the databases in response to commands from server 002.
[0041] The aforementioned terminal device 001, server 002, and database 003 can be interconnected via network 004. Specifically, any two of them can be interconnected, or the three can be connected in series.
[0042] Network 004 can be any type of network that can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. For example, network 004 can be a single network, which can be further subdivided into multiple sub-networks. Specifically, network 004 or the multiple sub-networks contained within it can be a Local Area Network (LAN), Ethernet, Token Ring, Wide Area Network (WAN), Internet, Virtual Network, Virtual Private Network (VPN), Intranet, Extranet, Public Switched Telephone Network (PSTN), Infrared Network, Wireless Network (e.g., Bluetooth, Wi-Fi), Mobile Network (e.g., 4G, 5G), Internet of Things (IoT), and / or any combination of these and / or other networks.
[0043] The technical solutions of this application can be applied to any application scenario requiring text retrieval, such as sensitive information screening, information querying, and data content quality inspection. It is understood that these application scenarios can all be... Figure 1 The implementation environment shown, or based on Figure 1 The implementation environment shown is simplified or extended to obtain the implementation environment.
[0044] Sensitive information screening is used to check whether target text contains sensitive information. For example, when a user uploads text information to the Internet or a database, sensitive information screening can identify whether the uploaded text information complies with regulations and whether it contains sensitive information.
[0045] Information retrieval is the most common data retrieval application scenario, used to retrieve data content of interest to users from databases. For example, in a text information retrieval scenario, when a user wants to find text sentences or text segments containing text of interest from massive amounts of text data, they can use the text of interest as a search condition and retrieve the text sentences or text segments containing the text of interest from the text database through text retrieval.
[0046] Data content quality inspection is similar to the sensitive information screening mentioned above. It primarily checks whether the target data contains necessary data items or data items that should not be included. For example, in text content quality inspection, one can determine whether the target text contains necessary or inappropriate text keywords by retrieving keywords that should or should not appear.
[0047] As a specific application scenario, the technical solution of this application embodiment can be applied to Figure 2 The illustration shows an application scenario for voice quality inspection. Voice quality inspection refers to checking whether a voice message contains necessary or prohibited content.
[0048] See Figure 2 As shown, the application scenarios for voice quality inspection can be based on Figure 1 The data retrieval system and speech recognition system shown in Figure 005 are implemented.
[0049] See Figure 2 As shown, the speech data of speaker 006 is recognized by speech recognition system 005 and converted into text content. The text content is processed by server 002 and then stored in database 003. Alternatively, the text content may be stored directly in database 003, either through server 002 or without server 002.
[0050] Speaker 006 can be a single speaker or two or more speakers. Correspondingly, the voice data input to the speech recognition system 005 can be single-person voice data or voice data from two- or multi-party dialogues.
[0051] In this application embodiment, the aforementioned voice quality inspection application scenario specifically refers to the application scenario of quality inspection of the voice during telephone communication between call center agents and customers, that is, in Figure 2In the application scenario shown, the speakers 006 specifically include the call center agent and the customer. The telephone voice data between the call center agent and the customer is input into the speech recognition system 005 for text conversion to obtain the text content of the telephone voice between the two parties. This text content is then stored in the database 003. Based on this, the user, through terminal device 001 and / or server 002, performs keyword retrieval on the telephone voice text between the call center agent and the customer stored in the database to determine whether the call center agent or the customer used inappropriate keywords or necessary keywords during the call. For example, it checks whether the call center agent used polite language or used impolite language when communicating with the customer.
[0052] In the above-mentioned scenarios of call center agents and customer voice quality inspection, voice quality inspection can also be carried out by role, such as inspecting only the voice of the call center agent, or only the voice of the customer, or inspecting both parties' voices at the same time.
[0053] To meet the aforementioned voice quality inspection requirements for different roles, this embodiment of the application includes at least the following settings in the text retrieval application interface of the terminal device 001: Figure 3 The input / output window shown.
[0054] See Figure 3 As shown, the text retrieval application interface includes at least three input / output windows: "Query Object," "Keywords," and "Context," as well as a "Search" button.
[0055] The "Query Target" window is used to set the voice quality inspection target, specifically "Agent," "Customer," and "Both Parties." The "Keywords" window is used to enter search keywords. The "Context" window displays the text search results obtained from the voice text of the query target using the keywords. The "Search" button triggers the search operation.
[0056] Accordingly, in order to enable separate retrieval of the voice text of different characters, the voice text of different characters is stored separately in database 003, that is, the voice text of different characters is stored separately, so as to facilitate text retrieval of the voice text of different characters.
[0057] For example, when a user selects "Agent" in the "Query Object" window, enters "Expired Now" in the "Keywords" window, and then clicks the "Search" button, a text search operation will begin. During the text search, the keyword "Expired Now" will be retrieved from the voice text generated by the agent during the telephone communication with the customer. When a text sentence containing the keyword "Expired Now" is found in the agent's voice text, this text sentence will be output from the "Context" window.
[0058] Below, we will provide an exemplary and detailed description of the text retrieval scheme proposed in the embodiments of this application, taking into account the implementation environment and application scenarios described above.
[0059] Exemplary methods
[0060] This application first proposes a text retrieval method, which can be executed by an electronic device, such as... Figure 1 The terminal device 001 and / or server 002 are shown. See also Figure 4 As shown, the method includes:
[0061] S101. Construct a regular expression based on the search keywords.
[0062] The aforementioned search keywords refer to the keywords used as search criteria for text retrieval from the target text. Specifically, these search keywords can be single text characters from any language, or phrases, short sentences, etc., composed of multiple text characters.
[0063] For example, suppose in Figure 2 In the application scenario shown, if a user wants to retrieve the text characters "expired now" from a target text, then "expired now" can be used as a search keyword. The user can input "expired now" into the search bar. Figure 3 In the "Keywords" window shown, you can select a query object to retrieve text sentences containing the keyword "expired now" from the text corresponding to the selected query object.
[0064] Regular expressions, also known as rule expressions, are logical formulas for manipulating strings. They can be used to retrieve text that matches a certain rule. Constructing a regular expression involves using predefined specific characters (including ordinary characters and metacharacters defined by computer languages) and combinations of these specific characters to form a "rule string." This "rule string" is used to express a filtering or selection logic for the string.
[0065] For example, the regular expression "^.{3,20}$" matches all characters of length 3-20; the regular expression "^[A-Za-z]+$" matches a string consisting of 26 English letters; the regular expression "^[AZ]+$" matches a string consisting of 26 uppercase English letters; the regular expression "^[A-Za-z0-9]+$" matches a string consisting of numbers and 26 English letters, and so on.
[0066] Given a regular expression and a string, the regular expression can be used to search or retrieve the string to determine whether the string conforms to the filtering logic of the regular expression, that is, whether it matches the regular expression. Alternatively, specific parts of the string that match the regular expression can be filtered out.
[0067] Because regular expressions are more flexible, logical, and functional, they can be used to quickly and easily filter or retrieve strings.
[0068] This application uses regular expressions for text retrieval. Specifically, when search keywords are determined or obtained, a regular expression is constructed from the search keywords. Specifically, the regular expression is constructed using all characters contained in the search keywords and preset characters. The constructed regular expression is used to retrieve the aforementioned search keywords from the target text, and to retrieve a preset string composed of the characters included in the search keywords and preset characters from the target text.
[0069] The aforementioned preset characters can be any one or more of preset punctuation marks and preset identifier characters, such as punctuation marks, graphic symbols, numbers, letters, spaces, punctuation marks, paragraph marks, etc., or identifier characters such as #, *, @, etc.
[0070] Based on the regular expression constructed above, as long as a certain string in the target text constitutes the above-mentioned search keyword, or the string contains all the characters of the above-mentioned search keyword and one or more of the preset characters, then the string matches the regular expression.
[0071] For example, if we construct a regular expression using the search keyword "expired now", then this regular expression can not only match the string "expired now", but also strings containing other characters such as "expired, now", "expired! now", and "expired. now".
[0072] Furthermore, given the flexibility of regular expressions, the matching tolerance can be set. For example, it can be set that only strings whose character order exactly matches the characters specified in the regular expression are considered a match; or, it can be set that any string containing the specified characters is considered a match, regardless of their order within the regular expression. Additionally, the length of the matched string can be limited; for example, the string matched by the regular expression cannot exceed a set character length.
[0073] In this embodiment of the application, a regular expression constructed based on the search keywords is used to filter search keywords from the target text, and to filter strings from the target text that contain all characters of the search keywords and one or more characters from preset characters, and the order of all characters of the search keywords is the same as the order of these characters in the search keywords, and the length of the string cannot exceed a set length, for example, it cannot exceed 2N-1, where N is the total number of characters contained in the search keywords.
[0074] If the string is too long, although it contains all the characters of the search keyword, it may already be a sentence with a meaning completely different from the search keyword. Alternatively, if the string length is increased to the level of a whole article, then due to character reuse, the article has a high probability of containing all the characters of the search keyword, but this is clearly not the ideal result for keyword retrieval. Therefore, this application's embodiments limit the length of the retrieved string. Practice has shown that if a special character is allowed between every two adjacent characters of the search keyword, a higher recall rate can be obtained compared to existing text retrieval schemes, and the reasonableness of the recall results can be guaranteed. If a special character is allowed between every two adjacent characters of the search keyword, the string length reaches 2N-1. Therefore, 2N-1 is used as the length limit for retrieving strings that match the regular expression from the target text.
[0075] If a string contains all the characters of the search keyword, but the order of the characters is not the same as the order of the characters in the search keyword, or its length exceeds 2N-1, then the string is not considered to match the regular expression.
[0076] It's understandable that directly searching text using keywords will only find strings that exactly match the keywords. However, by constructing regular expressions based on the keywords, it's possible not only to match strings that exactly match the keywords, but also to match strings containing all characters of the keywords, as well as other predefined characters—strings with a broader range of forms and content.
[0077] Therefore, using regular expressions as search criteria can make the search criteria more tolerant and generalized, and can effectively avoid the impact on the search results of search keywords due to the separation of search keywords by special characters.
[0078] S102. Retrieve the target string that matches the regular expression from the target text.
[0079] The target text mentioned above is the text to be searched, that is, the text that is expected to contain the search keywords mentioned above. Specifically, the target text can be text data of various forms, formats and contents, such as text sentences, text segments, text chapters, works, books, and periodicals in any language.
[0080] For example in Figure 2 In the application scenario shown, if a user wants to retrieve a specific keyword from the voice text of a call center agent, the user can... Figure 3 In the "Query Target" window shown, select "Agent". At this time, you can use the voice text of the agent as the target text and search for keywords from the voice text of the agent using this search method.
[0081] After determining the regular expression, it is matched against the characters of the target text to find strings that match the regular expression, which are then used as the target string. This target string can be the search keyword itself, or it can be a string containing all the characters of the search keyword, as well as one or more characters from a preset set of characters. Depending on the constraints of the regular expression, the length of the retrieved target string can be a set length or a string that does not exceed the set length.
[0082] In actual retrieval, a search method matching the database containing the target text can be selected for string retrieval. For example, in this embodiment, the text of telephone conversations between agents and customers is stored in an Elasticsearch database. The Elasticsearch database's match_phrase search method can then be used to search for the target text in the Elasticsearch database and find strings that match the regular expression.
[0083] S103. Determine the text sentence containing the target string from the target text.
[0084] Specifically, when the target string is retrieved from the target text after step S102, the text sentence containing the target string is extracted from the target text to obtain the text retrieval result corresponding to the above-mentioned search keywords.
[0085] For example, the text sentence containing the target string mentioned above can be a single text sentence or a text fragment composed of multiple consecutive single text sentences. Specifically, the single text sentence or text fragment mentioned above can be a single text sentence or text fragment with complete semantics obtained by dividing the target text according to its semantics, or it can be a single text sentence or text fragment obtained by formally dividing the target text into sentences or paragraphs.
[0086] For example, assuming the target text is pre-segmented or punctuated, the specific text content within the target text is itself stored after being divided into individual sentences or text fragments. In this case, if the target string is retrieved from the target text, the individual sentence or text fragment containing the target string can be identified as the text sentence containing the target string.
[0087] Specifically, if all characters of the retrieved target string are located within the same text sentence or text segment, then that text sentence or text segment is identified as the text sentence containing the target string. If the retrieved target string spans two adjacent text sentences or text segments—for example, some characters are located in the preceding text sentence or text segment, while other characters are located in the following text sentence or text segment adjacent to the preceding text sentence or text segment—then these two adjacent text sentences or text segments are jointly identified as the text sentence containing the target string.
[0088] Assuming the target text is unsegmented or unbroken, if the target string is retrieved from the target text, the target text is first divided into semantic or formal sentences or fragments. Then, the sentences or fragments containing the retrieved target string are identified as the final text sentences in the manner described above.
[0089] Following the example above, when a text sentence containing the retrieved target string is identified from the target text, that text sentence can be output as the search result corresponding to the search keyword.
[0090] As described above, the text retrieval method proposed in this application does not directly search the target text using the search keywords. Instead, it first constructs search keywords and a regular expression to retrieve a preset string composed of characters included in the search keywords and preset characters. Then, it retrieves the target string that matches the regular expression from the target text. Finally, it identifies the text sentence containing the retrieved target string from the target text, thus obtaining the search results corresponding to the search keywords.
[0091] Because regular expressions offer greater tolerance and flexibility for string retrieval, when applied to target text where adjacent characters belonging to the same word are separated by preset characters, this method can not only retrieve the target text containing the search keywords, but also, even if the search keywords are mixed with one or more preset characters, still retrieve the string containing both the search keywords and the preset characters, thus improving the recall rate for the search keywords. Applying this method to text retrieval scenarios with sentence breaks or paragraph breaks can avoid the impact of punctuation marks, paragraph breaks, or sentence breaks on the retrieval of search keywords, thereby significantly improving text retrieval performance.
[0092] As an example implementation, when constructing a regular expression based on search keywords, preset characters are first obtained, and then the regular expression is constructed based on all the characters of the search keywords and the obtained preset characters.
[0093] The aforementioned preset characters can be any one or more of preset punctuation marks and preset identifier characters, such as punctuation marks, graphic symbols, numbers, letters, spaces, punctuation marks, paragraph marks, etc., or identifier characters such as #, *, @, etc.
[0094] In this embodiment of the application, the characters included in the search keywords and preset characters are combined according to the construction rules of regular expressions to obtain regular expressions.
[0095] A regular expression constructed based on all characters of the search keyword and preset characters is used to search for the search keyword itself, as well as strings composed of one or more of the characters included in the search keyword and preset characters. In other words, even if the characters of the search keyword contain one or more of the preset characters, the regular expression can still be detected.
[0096] This regular expression can limit the number of preset characters that can be mixed into the search keywords. For example, if the search keywords contain no more than the set number of preset characters, the string composed of all the characters of the search keywords mixed with the preset characters will be matched by the regular expression. However, if the search keywords contain more than the set number of preset characters, the string composed of all the characters of the search keywords mixed with the preset characters will not be matched by the regular expression.
[0097] In addition, this regular expression can also limit the position of preset characters mixed in the characters of the search keywords. For example, it can only allow preset characters to be mixed between any two adjacent characters of the search keywords.
[0098] Practice has found that in actual text retrieval scenarios, special symbols in texts, especially punctuation marks, are the most frequently occurring characters apart from text characters, and are also the characters that have the greatest impact on text retrieval. For example, punctuation marks are essential characters for sentence segmentation and paragraph segmentation of texts. Therefore, in texts that have undergone sentence segmentation or paragraph segmentation, punctuation marks are the main character content that affects text retrieval.
[0099] In view of the above situation, the embodiments of the present application construct a regular expression according to all characters of a search keyword and preset punctuation marks.
[0100] The above-mentioned preset punctuation marks include all common punctuation marks, such as ",", ".", "?", "!", ";" and the like.
[0101] Based on all characters of the search keyword and the above-mentioned preset punctuation marks, the embodiments of the present application construct a regular expression, and define the occurrence positions of the preset punctuation marks in the search keyword in the regular expression.
[0102] Specifically, the regular expression constructed as described above is used to retrieve the above-mentioned search keyword, and is used to retrieve a character string that contains all characters of the above-mentioned search keyword and contains any preset punctuation mark between any two adjacent characters of the above-mentioned search keyword.
[0103] That is, if a certain character string in the target text only contains all characters of the search keyword, the character string will be retrieved and hit by the regular expression; if a certain character string in the target text not only contains all characters of the search keyword, but also includes a punctuation mark, and the punctuation mark is any one of the punctuation marks defined in the regular expression, and at the same time, the punctuation mark is located between any two adjacent characters of all the characters of the search keyword, the character string will also be retrieved and hit by the regular expression.
[0104] For example, assuming that the search keyword is "dao qi le xian zai" (the Chinese phrase meaning "it is due now"), a regular expression is constructed based on the characters "dao", "qi", "le", "xian", "zai" and each preset punctuation mark. The regular expression is used to retrieve the keyword "dao qi le xian zai", as well as character strings that contain any preset punctuation mark between any two adjacent characters of the keyword. For example, if a certain character string contains the characters "dao", "qi", "le", "xian", "zai", and any punctuation mark in the preset punctuation marks appears between "dao" and "qi", between "qi" and "le", between "le" and "xian", and between "xian" and "zai", the character string is still regarded as matching the regular expression.
[0105] As can be understood, the regular expression constructed in the above manner can not only retrieve the target keywords from the target text, but also retrieve strings containing both the target keywords and punctuation marks, even if the keywords are separated by punctuation marks in the target text. Therefore, using this regular expression to perform keyword retrieval tasks can avoid the influence of punctuation marks in the text on keyword retrieval, that is, it can avoid the impact of text segmentation and sentence breaks on document retrieval, thereby improving the retrieval recall rate for the search keywords and improving the retrieval effect.
[0106] As an exemplary implementation, see [link to relevant documentation]. Figure 5 As shown, the above-mentioned method of determining the text sentence containing the retrieved target string from the target text can be achieved by executing the following steps S1031 to S1033:
[0107] S1031. Determine the position information of the retrieved target string in the target text, and determine the position information of each individual sentence in the target text.
[0108] The position information of the target string in the target text refers to the position of the string as a whole in the target text.
[0109] Specifically, the position of the target string within the target text can be represented by the character position it occupies. For example, the position of the target string can be represented by the position from the first character to the last character of the target string. Here, the character position of a character in the target string refers to the position calculated starting from the first character of the target text.
[0110] Furthermore, the position of a target string within the target text can also be indicated by the sentence number it belongs to. Specifically, by sorting the sentences in the target text according to their order within the text, the sentence numbers can be determined. Based on this, when a target string is retrieved from the target text, determining the sentence number containing the target string also distinguishes the different target strings within the target text.
[0111] Furthermore, when two or more target strings appear in the same text sentence, if the position of the target string is represented only by the sequence number of the text sentence in which the target string is located, confusion may occur. Therefore, when multiple target strings appear in the same text sentence, the embodiments of this application distinguish the specific position of each target string in the text sentence by character position.
[0112] Therefore, determining the position information of the retrieved target string in the target text specifically includes determining the start and end positions of the retrieved target string in the target text, that is, the position of the first character of the target string and the position of the last character of the target string in the target text, and / or, determining the sequence number of the text sentence in which the retrieved target string is located. Furthermore, when the same text sentence contains multiple target strings, in addition to determining the sequence number of these target strings in the text sentence, the start and end positions of these target strings in the target text are also determined simultaneously, or the start and end positions of these target strings in the text sentence, in order to distinguish different target strings within the same text sentence.
[0113] Similarly, the position of each individual sentence in the target text refers to the position of the individual sentence as a whole within the target text.
[0114] When determining the positional information of each sentence in the target text, it is necessary to first determine the individual sentences in the target text, that is, to clarify the boundaries between each sentence.
[0115] For example, if the target text is pre-segmented or punctuated, the boundaries between individual sentences in the target text can be determined based on the segmentation or punctuation information, which means that individual sentences can be identified from the target text.
[0116] If the target text is not pre-segmented or punctuated, it can be divided into individual sentences by performing semantic segmentation or semantic punctuation on the target text.
[0117] As described above, the technical solutions of the embodiments of this application are applicable to Figure 2 The voice quality inspection application scenario shown is essentially an interactive scenario, specifically a voice interaction scenario. In this interactive scenario, the target text can be text obtained by performing speech-to-text processing on the dialogue speech generated in the target interactive scenario, or it can be text obtained by performing speech recognition on the dialogue speech generated in the target interactive scenario.
[0118] The interactive scenarios mentioned above can include telephone communication between call center agents and customers, voice communication between any two or more parties via telephone or the internet, or real-world scenarios where any two or more parties engage in dialogue.
[0119] The aforementioned dialogue audio refers to the audio of a conversation between at least two types of characters, that is, the dialogue audio includes the audio of both types of characters. These two types of characters can be categorized according to the dialogue initiator and the dialogue receiver, or they can be categorized according to their role identities; this embodiment does not impose any limitations.
[0120] The text obtained by processing the above-mentioned dialogue speech into text includes multiple text sentences, which can be further divided into multiple text sentences corresponding to the first type of character speech and multiple text sentences corresponding to the second type of character speech.
[0121] For example, assuming the target interaction scenario described above is a telephone communication scenario between a call center agent and a customer, then the target text described above is the text obtained by converting the telephone communication between the call center agent and the customer into text, which includes multiple text sentences corresponding to the call center agent's voice and multiple text sentences corresponding to the customer's voice.
[0122] In the above application scenario, as an optional implementation method for determining individual sentences in the target text, the individual sentences in the target text can be determined based on the speech recognition results obtained after segmentation of the dialogue speech in the above scenario, as well as the position information of each individual sentence in the target text.
[0123] Conventional speech recognition systems typically rely on Automatic Speech Recognition (ASR) technology to automatically identify speech. ASR inherently includes semantic sentence segmentation capabilities, thus its recognition results contain punctuation and sentence breaks. Therefore, the speech recognition result obtained by using such a system is a segmented speech recognition result, containing not only the target text content corresponding to the target speech but also the sentence segmentation information. Therefore, based on the segmented speech recognition result obtained from the target speech, it is possible to identify the individual sentences within the target text.
[0124] For example, the entire set of text characters contained in the speech recognition result obtained from the target speech is the entire set of text characters that make up the target text. The sentence segmentation information contained in the speech recognition result, such as punctuation marks and pause marks, reflects the sentence segmentation result of the spoken text, which is also the sentence segmentation result of the target text. By segmenting the target text according to the positions of punctuation marks and pause marks in the speech recognition result, individual sentences of the target text can be obtained.
[0125] For example, in this embodiment of the application, after performing speech recognition on the target speech, the text obtained from the speech recognition is stored in both whole sentence format and sentence segmentation format. The text stored in whole sentence format is used as the target text, and the speech quality inspection function can be realized by retrieving the target text.
[0126] Storing text as a whole sentence means storing the text as a single whole sentence; storing text as a segmented sentence means storing each segment of the text as a separate sentence.
[0127] In response to the aforementioned scenario of customer service representatives communicating with customers via telephone, this embodiment of the application uses the conversation between customer service representatives and customers as the target speech for speech recognition, converts the conversation into text, and then selects part or all of the recognized text as the target text.
[0128] In a preferred embodiment, the recognized text corresponding to the aforementioned dialogue speech is stored as a whole sentence, and also stored as a whole sentence by role. Therefore, when performing quality inspection on the aforementioned dialogue speech, the quality inspection can be performed on the speech text corresponding to a specific voice role, or on the speech text corresponding to all voice roles.
[0129] In the case where the aforementioned dialogue includes both first-type and second-type character voices, to facilitate the differentiation of voice texts from different voice characters, this application embodiment sets up a first-type text set, a second-type text set, and a third-type text set. The first-type text set stores the text corresponding to the voices of the first-type characters, the second-type text set stores the text corresponding to the voices of the second-type characters, and the third-type text set stores the text corresponding to the voices of the dialogue between the first-type and second-type characters—that is, it stores the text corresponding to the complete voice dialogue between the first-type and second-type characters.
[0130] As an exemplary implementation, by performing speech-to-text processing on the aforementioned dialogue speech, the aforementioned first type of text set, second type of text set, and third type of text set can be generated as follows:
[0131] First, the above-mentioned dialogue speech is processed by speech recognition to obtain the speech recognition results of sentence segmentation, and at least each text sentence in the speech recognition results, as well as the speech role and position information corresponding to each text sentence, are stored in the speech text set.
[0132] As described above, this embodiment of the application uses an ASR-based speech recognition system to perform speech recognition on the aforementioned dialogue speech, obtaining a speech recognition result. This speech recognition result is the text content after sentence segmentation, including the characters corresponding to the speech and punctuation marks.
[0133] The speech recognition results described above are stored to form a speech-text set. This speech-text set stores each individual text sentence from the speech recognition results, as well as the attribute information for each individual text sentence. This attribute information includes at least the voice actor corresponding to the individual text sentence and the position information of the individual text sentence.
[0134] This application embodiment stores the speech recognition results of sentence segmentation in a formatted manner. For example, each text sentence is stored as a JSON string. The JSON string contains information such as the start time (beginTime), end time (endTime), voice channel (channelId), emotion type (emotionType), emotion level (emotionValue), noise, speaker role (role), silence duration (silenceDuration), voice content (text), and volume (volume) of the text sentence in the target speech.
[0135] Specifically, it can be stored in the following manner:
[0136] {"change_asr_text":
[0137] "[{"beginTime":2700,"channelId":0,"emotionType":"M","emotionValue":0.1,"endTime":3760,"noise":false,"role":"1","silenceDuration":2700,"text":"Hey! ","volume":67.78014601428619},
[0138] {"beginTime":3240,"channelId":1,"emotionType":"N","emotionValue":0.2,"endTime":5380,"noise":false,"role":"0","silenceDuration":0,"text":"Hello, is this Zhang Moumou?","volume":78.37947800427285},
[0139] {"beginTime":6340,"channelId":1,"emotionType":"N","emotionValue":0.3,"endTime":10380,"noise":false,"role":"0","silenceDuration":960,"text":"XX's repayment period has expired, and now X thousand X hundred XXX yuan has been processed by the APP today.","volume":77.03651663572252},
[0140] {"beginTime":11680,"channelId":1,"emotionType":"N","emotionValue":0.3,"endTime":15328,"noise":false,"role":"0","silenceDuration":1300,"text":"If you repay before noon, you will be required to pay off the entire amount plus a penalty.","volume":80.46160440715954}]"
[0141] }
[0142] It is understandable that the above-mentioned speech-text set can be obtained by formatting and storing the speech recognition results of the segmented sentences. Through this speech-text set, the individual text sentences obtained from the speech recognition of the target speech can be identified intuitively and accurately, as well as the start time, end time, corresponding speaker, and other attribute information of each individual text sentence.
[0143] In addition, this application embodiment also stores the text of the speech recognition result of the target speech in whole sentences according to roles, so as to obtain a text set corresponding to different roles.
[0144] Based on the individual text sentences in the aforementioned speech-text set, and the speech role information of each individual text sentence in the speech-text set, the individual text sentences corresponding to the speech of the first type of role are extracted from the speech-text set, combined into a complete sentence, and then stored to obtain the first type of text set; the individual text sentences corresponding to the speech of the second type of role are extracted from the speech-text set, combined into a complete sentence, and then stored to obtain the second type of text set; and the individual text sentences corresponding to the speech of the first type of role and the speech of the second type of role are extracted from the speech-text set, combined into a complete sentence, and then stored to obtain the third type of text set.
[0145] Taking the aforementioned scenario of quality inspection of telephone voice communication between a call center agent and a customer as an example, since this voice quality inspection scenario involves the voices of both the call center agent and the customer, the corresponding voice text content for each party will be obtained through voice recognition. To facilitate the differentiation of voice text content from different speakers, this embodiment of the application stores the voice recognition results as complete sentences divided by role, specifically into two parties (asr_both_text), the call center agent (asr_service_text), and the customer (asr_customer_text), as follows:
[0146] {"asr_both_text":"Hello! Is this Zhang Moumou? The repayment deadline for XX is approaching. The APP processed X thousand X hundred XXX yuan today. Please make the payment yourself before noon. Otherwise, the loan will be fully repaid plus a late payment penalty."}
[0147] {"asr_service_text":"Hello, is this Mr. / Ms. Zhang? Your repayment deadline for XX is approaching. The app processed your payment of X thousand X hundred XXX yuan today. Please make the payment yourself before noon. Otherwise, you will have to pay off the entire amount plus a late payment penalty."}
[0148] {"asr_customer_text":"Hey!"}
[0149] The text corresponding to both parties (asr_both_text) is the third type of text set mentioned above; the text corresponding to the agent (asr_service_text) and the customer (asr_customer_text) is the first type of text set and the second type of text set mentioned above.
[0150] This application embodiment stores the text content of the speech recognition results of the dialogue speech by role, which facilitates the quality inspection of the speech of different roles in the later stage. That is, by selecting the speech text content of the corresponding role from the stored text of the speech recognition results, the speech quality inspection of that role can be realized.
[0151] Based on the aforementioned speech-text set, as well as the first, second, and third text sets, when performing quality inspection on the speech of a single character or both characters, the speech recognition result text corresponding to that character or both characters can be determined as the target text for text retrieval. That is, the text from any one of the aforementioned first, second, and third text sets is selected as the target text. Then, by mapping and matching the text in the selected target text with the individual sentences in the aforementioned speech-text set, the target text can be divided into individual sentences.
[0152] Specifically, based on the "role" attribute of each JSON string in the aforementioned voice text set, the voice text sentences of the corresponding role to the target text are determined. The determined voice text sentences are then mapped and matched with the text in the target text. The text that matches each voice text sentence is found in the target text, thus obtaining the individual text sentences contained in the target text.
[0153] For example, assuming a user selects to perform quality inspection on the agent's voice, the text corresponding to the agent's voice will be used as the target text for text retrieval, i.e., "Hello, is this Zhang Moumou? The repayment period for XX is due, and the APP is processing X thousand X hundred XXX yuan today. Please repay it before noon, otherwise the loan will be fully repaid plus a penalty." Then, based on the "role" attribute of each JSON string in "change_asr_text", the corresponding agent's text sentence will be determined, i.e., "Hello, is this Zhang Moumou?" and "The repayment period for XX is due, and the APP is processing X thousand X hundred XXX yuan today." The text "Hello, Mr. / Ms. Zhang? Your XX repayment deadline has passed. The APP is processing your X thousand X hundred XXX yuan today. Please repay before noon. Otherwise, you will be charged a penalty fee." consists of three individual text sentences: "Hello, Mr. / Ms. Zhang?", "Your XX repayment deadline has passed. The APP is processing your X thousand X hundred XXX yuan today. Please repay before noon. Otherwise, you will be charged a penalty fee."
[0154] After identifying each individual sentence in the target text, the positional information of each individual sentence within the target text can be further determined.
[0155] Specifically, the position of a single text sentence within the target text can be represented by the character position it occupies. For example, the position of a single text sentence within the target text can be represented by the character position from the first character of the sentence to the character position of the last character of the sentence. Here, the character position of a single text sentence refers to the character position calculated starting from the first character of the target text.
[0156] Furthermore, the position of a sentence within the target text can also be indicated by its index. Specifically, by sorting the sentences in the target text according to their order within the target text, the index of each sentence can be determined.
[0157] Therefore, determining the position information of each text sentence in the target text specifically includes determining the start and end positions of each text sentence in the target text, that is, the character position of the first character of each text sentence in the target text, and the character position of the last character of each text sentence in the target text, and / or, determining the sequence number of each text sentence in the target text.
[0158] As an exemplary implementation, in a pre-segmented or sentence-divided target text, by sorting characters or text sentences in chronological order, the character number and text sentence number of each character and text sentence can be determined. Both the character number and the text sentence number can serve as positional representations of the characters and text sentences.
[0159] Based on this, when a target string is retrieved from the target text, the character position of the target string or the sentence position of the target string within a single sentence of the text can be determined first. Then, the position information of the target string can be determined by using the character number corresponding to that character position or the sentence number corresponding to that sentence position.
[0160] S1032. Based on the position information of the target string in the target text and the position information of each text sentence in the target text, determine the target text sentence from the target text.
[0161] The aforementioned target text sentence refers to the text sentence that is matched by the target string retrieved from the target text. A text sentence being matched by the target string means that one or more characters of the target string are present in that text sentence.
[0162] Since the target string is a string consisting of consecutive characters, there are two main cases when the target string matches a single sentence in the text:
[0163] One type is where all characters of the target string are within a single text sentence, meaning the target string is part of a single text sentence. For example, suppose we search for the keyword "expired now" in the target text "Hello, Mr. / Ms. Zhang, right? The repayment period for XX has expired. Now, X thousand X hundred XXX yuan is being processed by the APP today. I will repay it before noon. If I don't repay it, I will have to pay off the loan plus a penalty." According to the above-described scheme in this application embodiment, the target string "expired now" is retrieved. Moreover, "The repayment period for XX has expired. Now, X thousand X hundred XXX yuan is being processed by the APP today." is a single text sentence. Therefore, we can determine that all characters of the target string "expired now" are within this single text sentence. Thus, the single text sentence "The repayment period for XX has expired. Now, X thousand X hundred XXX yuan is being processed by the APP today." is the target text sentence.
[0164] Another scenario is where a portion of the target string is within a single text sentence; that is, a part of the target string is part of a single text sentence. This typically occurs when a portion of the target string is within one text sentence, while another portion is within an adjacent text sentence. In this case, the target string spans and covers two adjacent text sentences, and both adjacent text sentences are considered part of the target text sentence.
[0165] For example, suppose we search for the keyword "expired now" in the target text "Hello, Mr. / Ms. Zhang, right? The repayment period for XX has expired. Now, the APP is processing X thousand X hundred XXX yuan today. If you repay it before noon, you will have to pay off the entire amount plus a penalty." According to the above-described scheme in the embodiments of this application, the target string "expired now" is retrieved. Moreover, both "The repayment period for XX has expired." and "Now, the APP is processing X thousand X hundred XXX yuan today" are single text sentences. Therefore, it can be determined that the target string "expired now" simultaneously matches both the single text sentences "The repayment period for XX has expired." and "Now, the APP is processing X thousand X hundred XXX yuan today." Therefore, both the single text sentences "The repayment period for XX has expired." and "Now, the APP is processing X thousand X hundred XXX yuan today" are target text sentences.
[0166] Based on the above, it can be understood that if the position of the target string in the target text intersects with the position of a certain sentence in the target text, then at least one character of the target string coincides with that sentence. In this case, the target string must match that sentence, and that sentence must also be the target sentence.
[0167] Therefore, as an optional implementation, this application embodiment determines, based on the position information of the target string in the target text and the position information of each text sentence in the target text, the text sentence whose position intersects with the position of the target string, and takes it as the target text sentence.
[0168] For example, if the position information of the target string contains the sequence number of the text sentence in which the target string is located, and the position information of each text sentence in the target text contains the sequence number of each text sentence, then the text sentence corresponding to the sequence number of the text sentence in which the target string is located is the text sentence that intersects with the position of the target string, and this text sentence is the target text sentence.
[0169] If the position information of the target string includes the start and end positions of the target string, and the position information of each sentence in the target text includes the start and end positions of each sentence, then by determining whether there is an overlap between the start and end positions of the target string and the start and end positions of each sentence in the target text, it can be determined whether there is an intersection between the position of the target string and the position of each sentence in the target text. If there is an intersection, then the sentence that intersects with the position of the target string can be identified as the target text sentence.
[0170] After the above steps S1031 and S1032, the target text sentence is determined from the target text based on the target string retrieved from the target text.
[0171] S1033. Based on the determined target text sentence, construct a text sentence containing the retrieved target string.
[0172] As described above, the target text sentences identified through the above processing are text sentences that include some or all of the characters of the target string. These text sentences all contain the search keyword characters; therefore, these text sentences, or combinations of these text sentences, can be used as search results obtained by retrieving the aforementioned search keywords from the target text.
[0173] In this embodiment of the application, after determining the target text sentences from the target text through the above steps S1031 and S1032, each determined target text sentence is designated as a text sentence containing the target string, and / or a text sentence obtained by combining the determined target text sentences is designated as a text sentence containing the retrieved target string. These text sentences can then be output as search results corresponding to the search keywords. Therefore, the final determined text sentence containing the retrieved target string can be one or more text sentences.
[0174] As an exemplary implementation, this application embodiment determines the determined target text sentences directly as text sentences containing the retrieved target strings based on the determined positional relationships of each target text sentence, or combines the determined target text sentences as text sentences containing the retrieved target text strings.
[0175] For example, when there are multiple target text sentences, a text sentence containing the retrieved target string can be constructed based on the identified target text sentences as follows:
[0176] For each identified target text sentence, traverse each target text sentence sequentially according to its position order in the target text.
[0177] When traversing to the i-th target text sentence, if the i-th target text sentence has adjacent target text sentences with consecutive positional information, then the i-th target text sentence is concatenated with its adjacent target text sentences to obtain a text sentence containing the target string. Here, i is a positive integer starting from 1.
[0178] The above-mentioned concatenation process specifically involves concatenating adjacent target text sentences according to their relative positions within the target text to obtain a concatenated text sentence, which serves as the text sentence containing the retrieved target string.
[0179] If the i-th target text sentence does not have any adjacent target text sentences with consecutive positional information, then the i-th target text sentence is regarded as a text sentence containing the target string.
[0180] Based on the above processing, when outputting text sentences containing the retrieved target string, each text sentence can be output separately, for example, different text sentences can be output in segments. This allows users to more intuitively determine which text sentences contain the search keywords through the search results.
[0181] As can be seen, the text retrieval method proposed in this application can recall text sentences containing all characters of the search keywords from the target text, thus making the retrieval of search keywords more flexible and achieving a higher recall rate. At the same time, this method can effectively filter text content in the target text that is unrelated to the search keywords, making the search results more intuitive and concise, and greatly improving the readability of the search results.
[0182] Figure 5 Steps S101 and S102 in the method embodiment shown are respectively with Figure 4 The steps S101 and S102 in the method embodiment shown correspond to each other. For details of their processing, please refer to the documentation. Figure 4 The corresponding implementation examples are described below, and will not be repeated here.
[0183] As an optional implementation method, see [link to implementation details]. Figure 6 As shown, the document retrieval method proposed in this application embodiment further includes step S104:
[0184] S104. Output each text sentence and highlight the target string in each text sentence.
[0185] Setting the target string in the text sentence to highlight means setting the target string in the text sentence to be displayed differently from other characters in the text sentence.
[0186] For example, to highlight a target string in a text sentence, you can set the target string in the text sentence to be displayed with different font, font size, color and other display attributes than other strings, or you can add a marker to the target string in the text sentence, such as adding an underline, strikethrough, text box, etc.
[0187] In this embodiment of the application, after constructing a regular expression, retrieving the target text based on the regular expression, and determining the text sentences containing the target string based on the retrieved target string, each determined text sentence is output. At the same time, when outputting each text sentence, the target string in each text sentence is highlighted.
[0188] According to the above text output method, users can not only intuitively understand which text sentences in the target text contain the search keywords, but also identify the specific location and form of the search keywords in these text sentences by highlighting them.
[0189] For example, suppose a user wants to retrieve the keyword "expired now" from the target text "Hello, Mr. / Ms. Zhang, right? The repayment period for XX is due. The APP is processing X thousand X hundred XXX yuan today. I will repay it before noon. If I don't repay, I will have to pay off the entire amount plus a penalty." According to the above embodiments of this application, a regular expression is first constructed based on "expired now." Then, this regular expression is used to retrieve the target string matching the regular expression from the target text, namely "expired now." The text sentence containing the retrieved target string is then determined, namely "The repayment period for XX is due. The APP is processing X thousand X hundred XXX yuan today." Finally, this text sentence is output, and the target string "expired now" is extracted and displayed. For example, adding an underline to the target string "expired now" results in the final output text sentence "XX repayment period..." It's expired now. The text "X thousand X hundred XXX yuan was processed well by the APP today." indicates that, through this text sentence, users can intuitively determine which text sentence in the target text contains the keyword "expired now." Furthermore, they can intuitively determine the form in which the keyword "expired now" exists within this text sentence, namely, that the keyword "expired now" is separated by commas ",".
[0190] As an exemplary implementation, this application embodiment adds an identifier tag to the target string in the output text sentence, so that the target string in the text sentence is highlighted when the output is displayed.
[0191] Specifically, in this embodiment of the application, each text sentence is output through the following steps A1-A2, and the target string in each output text sentence is highlighted:
[0192] A1. Use the regular expression as a search condition to locate the target string from each text sentence.
[0193] Specifically, the regular expression mentioned above is the regular expression constructed based on the search keywords in step S101 of the above embodiment.
[0194] Since the final output text sentence may be obtained by concatenating and combining individual text sentences from the target text, in order to accurately filter out the target string in the output text sentence, this application embodiment again uses the above-mentioned regular expression to search each text sentence to be output, and locates each target string from it.
[0195] A2. Add an identifier tag to the target string located in each text sentence.
[0196] Specifically, after locating each target string from each text sentence to be output, a label is added to each target string. This label includes, but is not limited to, highlight labels, attribute labels for adjusting display properties such as font, font size, and color, or labels for adding underlines, strikethroughs, text boxes, etc.
[0197] For example, assuming the target string "expired, now" is located in the text to be output, a color label is added to the string "expired, now". <span style='color:red'>"," ".
[0198] After the above labeling process, each text sentence is output. Since the target strings in the output text sentences are all strings carrying display color labels, when these strings are rendered and displayed, they will be displayed in the color indicated by the label.
[0199] Figure 6 For details of steps S101 to S103 in the illustrated embodiment, please refer to [link / reference needed]. Figure 1 The steps S101 to S103 in the illustrated embodiment are described below.
[0200] It should also be noted that the above embodiments of this application describe the text retrieval process using a single search keyword as an example. In practical applications of the technical solutions of these embodiments, users may wish to retrieve multiple keywords from the target text simultaneously. The technical solutions of the above embodiments of this application also support this scenario.
[0201] When a user enters multiple search keywords, as described in the above embodiments, a regular expression is constructed for each search keyword. Then, the target text is searched according to the constructed regular expression to obtain the target string. Based on the obtained target string, the text sentences containing the retrieved target string are determined and output.
[0202] After determining the text sentence for each search keyword, the text sentences corresponding to each search keyword can be grouped together, adjacent text sentences can be merged, and the merged text sentence can be output. Alternatively, the text sentence corresponding to each search keyword can be output separately.
[0203] The regular expression construction, target string retrieval, text sentence determination, and text sentence merging in the above processing can all be referred to the descriptions of the above embodiments, and will not be detailed here.
[0204] To more vividly illustrate the processing of the technical solutions of this application in practical application scenarios, the specific text retrieval process will be introduced below with specific examples.
[0205] Referring to the example above, suppose a user wants to perform voice quality inspection on telephone communications between call center agents and customers. See also... Figure 2 As shown, the voice data of telephone communication between the call center agent and the customer has been converted into text data and stored through the voice recognition system. Therefore, the voice quality inspection can be achieved through text retrieval.
[0206] Text data of telephone conversations between agents and customers is stored in [location]. Figure 2 The database shown has two storage methods, as described in the above embodiments: one method is to store each sentence in JSON string format according to the sentence segmentation recognition results of speech recognition; the other method is to store the text content of speech recognition by role.
[0207] This text retrieval process can be achieved through user operations on the client side and background processing by the system in response to these user operations. Here, the "client side" refers to the client on which the user performs the text retrieval operation; for example, it could be... Figure 2 The terminal device 001 shown can display a text search application interface on the terminal side. This text search application interface can be as follows: Figure 3 The text search application interface shown is shown. Users can enter search keywords, search scope, and other information in this text search application interface. Users can also trigger the system to perform corresponding text search processing by triggering the search button in the text search application interface.
[0208] The aforementioned system side refers to the device or system that actually performs the text retrieval process; for example, it could be... Figure 2 The server 002 shown is either a combination of server 002 and database 003, or the local system of terminal device 001.
[0209] Suppose a user wants to inspect the quality of a caller's voice messages to see if the caller used the keywords "expired now" and "760".
[0210] See Figure 7 As shown, during the actual search, the user selects "Agent" in the "Query Object" window of the text search application interface on the client side, and then enters "Expired now, 760" in the "Keywords" window. The comma "," between the two keywords is used to separate them. Then the user clicks the "Search" button on the text search application interface.
[0211] At this point, the query object information and search keyword information are transmitted to the system side.
[0212] The system first constructs regular expressions based on each search keyword.
[0213] Secondly, the constructed regular expressions are used to search the text of the agent's voice, and the target string that matches each regular expression is obtained.
[0214] That is, the target string to be matched by the regular expression is retrieved and constructed from the sentence "Hello, Mr. / Ms. Zhang, right? The repayment period for XX has expired. Now, the APP has processed X thousand X hundred XXX yuan. Please repay it before noon. Otherwise, the loan will be fully repaid plus a penalty."
[0215] As can be understood from the above embodiments of this application, the target strings "expired, now" and "760" can be retrieved.
[0216] Next, the target text sentences that match each target string are identified from the text of the agent's speech.
[0217] As can be seen from the above search, the two target strings "expired, now" and "760" both matched the same text sentence "XX repayment deadline has expired, now X thousand X hundred XXX yuan. The APP is handling it well today." Therefore, this text sentence is the target text sentence.
[0218] Then, adjacent sentences in each target text are merged.
[0219] Finally, the merged text sentences are output.
[0220] Since the target text sentence identified by searching for the keywords "expired now" and "760" is the same text sentence, the above-mentioned merging of adjacent text sentences will still result in the same text sentence.
[0221] Specifically, when outputting the merged text sentences, a display color label (e.g., a red display label) is added to the target string in the output text sentence to highlight the target string.
[0222] Each text sentence output by the system is rendered and displayed on the client side, ultimately appearing in the "Context" window of the text retrieval application interface on the client side. When displaying the received text sentences on the client side, because the strings "due, now" and "760" in the text sentence carry display color labels, these two strings are displayed according to the color labels. That is, the final text displayed in the "Context" window of the text retrieval application interface on the client side is "XX repayment deadline has expired, now X thousand X hundred XXX yuan, today the APP processed it well." Furthermore, the characters "due, now" and "760" are displayed in a different color to distinguish them from other characters; for example, these characters are displayed in red, while other characters are displayed in black.
[0223] Exemplary device
[0224] Corresponding to the text retrieval method described above, this application also provides a text retrieval device, see [link to relevant documentation]. Figure 8 As shown, the device includes:
[0225] The retrieval preparation unit 100 is used to construct a regular expression based on the retrieval keywords; wherein, the regular expression is used to retrieve the retrieval keywords and a preset string, the preset string being composed of characters included in the retrieval keywords and preset characters; in the preset string, the preset characters are located between any two adjacent characters in the retrieval keywords;
[0226] The text retrieval unit 110 is used to retrieve a target string that matches the regular expression from the target text; the target text has the characteristic that two adjacent characters belonging to the same word are separated by the preset character;
[0227] The retrieval processing unit 120 is used to determine the text sentence containing the target string from the target text.
[0228] As an optional implementation, the step of constructing a regular expression based on the search keywords includes: combining the characters included in the search keywords and preset characters according to the construction rules of regular expressions to obtain a regular expression.
[0229] As an optional implementation, the preset characters include any one or more preset punctuation marks and preset identifier characters.
[0230] As an optional implementation, the target text includes multiple text sentences, and determining the text sentence containing the target string from the target text includes: determining the target text sentence from the target text based on the retrieved target string, wherein there is an intersection between the position of the target text sentence in the target text and the position of the target string in the target text; and constructing a text sentence containing the target string based on the determined target text sentence.
[0231] As an optional implementation, the position information of the target string in the target text includes: the start position and end position of the target string in the target text, and / or the sequence number of the text sentence in which the target string is located; the position information of each text sentence in the target text includes: the start position and end position of each text sentence in the target text, and / or the sequence number of each text sentence in the target text.
[0232] As an optional implementation, the target text is text from any one of three sets: a first set, a second set, and a third set, generated by speech-to-text processing of the dialogue speech generated in the target interactive scene. The dialogue speech includes first-type character speech and second-type character speech; the first set of texts includes text corresponding to the first-type character speech, the second set of texts includes text corresponding to the second-type character speech, and the third set of texts includes text corresponding to the dialogue speech. The first set of texts, the second set of texts, and the third set of texts are obtained through the following processing: performing speech recognition processing on the dialogue speech to obtain segmented sentences. The speech recognition results are processed, and at least each text sentence in the speech recognition results, along with the corresponding voice role and location information for each text sentence, are stored in a speech text set. Multiple text sentences corresponding to the voice of a first type of role are extracted from the speech text set and added to the first type of text set. Multiple text sentences corresponding to the voice of a second type of role are extracted from the speech text set and added to the second type of text set. Furthermore, multiple text sentences corresponding to the voice of the first type of role and the voice of the second type of role are extracted from the speech text set and added to the third type of text set. Each text sentence in the target text is determined by mapping and matching the text in the target text with each text sentence in the speech text set.
[0233] As an optional implementation, the step of constructing a text sentence containing the retrieved target string based on the determined target text sentences includes: determining each determined target text sentence as a text sentence containing the target string, and / or determining the text sentence obtained by combining the determined target text sentences as a text sentence containing the target string.
[0234] As an optional implementation, the number of target text sentences is multiple; the step of constructing a text sentence containing the target string based on the determined target text sentences includes: sequentially traversing multiple target text sentences; for the i-th target text sentence, if the i-th target text sentence has adjacent target text sentences with consecutive positional information, then concatenating the i-th target text sentence with the adjacent target text sentences of the i-th text sentence, and obtaining a text sentence containing the target string after concatenation; where i is a positive integer; if the i-th target text sentence does not have adjacent target text sentences with consecutive positional information, then the i-th target text sentence is taken as a text sentence containing the target string.
[0235] As an optional implementation, the retrieval processing unit is further configured to: use the regular expression as a retrieval condition to locate the target string from each text sentence; and add an identifier tag to the target string located in each text sentence.
[0236] The text retrieval device provided in this embodiment belongs to the same concept as the text retrieval method provided in the above embodiments of this application. It can execute the text retrieval method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of executing the text retrieval method. Technical details not described in detail in this embodiment can be found in the specific processing content of the text retrieval method provided in the above embodiments of this application, and will not be repeated here.
[0237] Exemplary electronic devices
[0238] Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 9 As shown, the electronic device includes a memory 200 and a processor 210; wherein the memory 200 is connected to the processor 210 and is used to store computer programs; the processor 210 is used to implement the text retrieval method disclosed in any of the above embodiments by running the computer programs stored in the memory 200.
[0239] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.
[0240] Processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. The bus may include a pathway for transmitting information between various components of the computer system. Processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It may also be a digital signal processor (DSP), application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 210 may include a main processor, and may also include a baseband chip, modem, etc. Memory 200 stores the computer program that executes the technical solution of the present invention, and may also store the operating system and other key business functions. Specifically, the computer program may include program code, which includes computer operation instructions. More specifically, memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc. Input device 230 may include means for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor, etc. Output device 240 may include means for allowing information to be output to the user, such as a display screen, printer, speaker, etc. Communication interface 220 may include means using any transceiver to communicate with other devices or communication networks, such as Ethernet, wireless access network (RAN), wireless local area network (WLAN), etc. Processor 210 executes the program stored in memory 200 and calls other devices, and can be used to implement the various steps of any of the text retrieval methods provided in the above embodiments of this application.
[0241] Exemplary computer program products and storage media
[0242] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the text retrieval method described in the "Exemplary Methods" section of this specification.
[0243] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this application. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0244] Furthermore, embodiments of this application may also be storage media storing a computer program thereon, the computer program being executed by a processor of the steps in the text retrieval method described in the "Exemplary Methods" section above this specification.
[0245] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0246] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The steps in the methods of each embodiment of this application can be adjusted, merged, or deleted according to actual needs, and the technical features described in each embodiment can be replaced or combined. The modules and sub-modules in the devices and terminals of each embodiment of this application can be merged, divided, and deleted according to actual needs.
[0247] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0248] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method of text retrieval, characterized by, include: A regular expression is constructed based on the search keywords; wherein, the regular expression is used to search for the search keywords and a preset string, the preset string being composed of characters included in the search keywords and preset characters; in the preset string, the preset characters are located between any two adjacent characters in the search keywords; Retrieve target strings from target text that match the regular expression; the target text has the characteristic that two adjacent characters belonging to the same word are separated by the preset character; Identify the text sentence containing the target string from the target text; The target text is text from any one of the three sets of text: the first set, the second set, and the third set. The first type of text set, the second type of text set, and the third type of text set are generated by speech-to-text processing of the dialogue voice generated in the target interaction scenario; the dialogue voice includes the voice of the first type of character and the voice of the second type of character, the first type of text set includes the text corresponding to the voice of the first type of character, the second type of text set includes the text corresponding to the voice of the second type of character, and the third type of text set includes the text corresponding to the dialogue voice. The first type of text set, the second type of text set, and the third type of text set are obtained through the following processing: The dialogue speech is processed by speech recognition to obtain the speech recognition result of sentence segmentation, and at least each text sentence in the speech recognition result, as well as the voice role and position information corresponding to each text sentence, are stored in the speech text set. Extract multiple text sentences corresponding to the voices of the first type of character from the voice text set and add them to the first type of text set; extract multiple text sentences corresponding to the voices of the second type of character from the voice text set and add them to the second type of text set; and extract multiple text sentences corresponding to the voices of the first type of character and the second type of character from the voice text set and add them to the third type of text set.
2. The method of claim 1, wherein, The step of constructing a regular expression based on search keywords includes: According to the construction rules of regular expressions, the characters included in the search keywords and preset characters are combined to obtain a regular expression.
3. The method of claim 2, wherein, The preset characters include any one or more preset punctuation marks and preset identifier characters.
4. The method of claim 1, wherein, The target text includes multiple single text sentences, and determining the text sentence containing the target string from the target text includes: Based on the retrieved target string, a target text sentence is determined from the target text, wherein there is an intersection between the position of the target text sentence and the position of the target string in the target text; Based on the identified target text sentence, a text sentence containing the target string is constructed.
5. The method of claim 4, wherein, The position information of the target string in the target text includes: the start position and end position of the target string in the target text, and / or the sequence number of the text sentence in which the target string is located; the position information of each text sentence in the target text includes: the start position and end position of each text sentence in the target text, and / or the sequence number of each text sentence in the target text.
6. The method of claim 4, wherein, The step of constructing a text sentence containing the retrieved target string based on the determined target text sentence includes: Each identified target text sentence is determined as a text sentence containing the target string, and / or, the text sentence obtained by combining the identified target text sentences is determined as a text sentence containing the target string.
7. The method of claim 4, wherein, The number of target text sentences is multiple; the step of constructing a text sentence containing the target string based on the determined target text sentences includes: Multiple target text sentences are traversed sequentially. For the i-th target text sentence, if there are adjacent target text sentences with continuous position information, the i-th target text sentence is concatenated with its adjacent target text sentences to obtain a text sentence containing the target string; where i is a positive integer. If the i-th target text sentence does not have any adjacent target text sentences with consecutive positional information, then the i-th target text sentence is regarded as a text sentence containing the target string.
8. The method of claim 1, wherein, The method further includes: The regular expression is used as a search condition to locate the target string from each text sentence; Add an identifier tag to the target string located in each text sentence.
9. A text retrieval apparatus characterized by comprising: include: The retrieval preparation unit is used to construct a regular expression based on the retrieval keywords; wherein, the regular expression is used to retrieve the retrieval keywords and a preset string, the preset string being composed of characters included in the retrieval keywords and preset characters; in the preset string, the preset characters are located between any two adjacent characters in the retrieval keywords; A text retrieval unit is used to retrieve target strings that match the regular expression from target text; the target text has the characteristic that two adjacent characters belonging to the same word are separated by the preset character; A retrieval processing unit is used to determine, from the target text, a text sentence containing the target string; The target text is text from any one of the three sets of text: the first set, the second set, and the third set. The first type of text set, the second type of text set, and the third type of text set are generated by speech-to-text processing of the dialogue voice generated in the target interaction scenario; the dialogue voice includes the voice of the first type of character and the voice of the second type of character, the first type of text set includes the text corresponding to the voice of the first type of character, the second type of text set includes the text corresponding to the voice of the second type of character, and the third type of text set includes the text corresponding to the dialogue voice. The first type of text set, the second type of text set, and the third type of text set are obtained through the following processing: The dialogue speech is processed by speech recognition to obtain the speech recognition result of sentence segmentation, and at least each text sentence in the speech recognition result, as well as the voice role and position information corresponding to each text sentence, are stored in the speech text set. Extract multiple text sentences corresponding to the voices of the first type of character from the voice text set and add them to the first type of text set; extract multiple text sentences corresponding to the voices of the second type of character from the voice text set and add them to the second type of text set; and extract multiple text sentences corresponding to the voices of the first type of character and the second type of character from the voice text set and add them to the third type of text set.
10. An electronic device, comprising: include: Memory and processor; The memory is connected to the processor and is used to store computer programs; The processor is configured to implement the text retrieval method as described in any one of claims 1 to 8 by running a computer program in the memory.
11. A storage medium, characterized by The storage medium stores a computer program, which, when executed by a processor, implements the text retrieval method as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Content retrieval method and a system
CN109284362A
