Unstructured text processing method and device, equipment, storage medium and program product

Abstract

The invention provides an unstructured text processing method which can be applied to the technical field of artificial intelligence. The unstructured text processing method comprises the following steps: acquiring an initial text to be processed, wherein the initial text is an unstructured text; based on a preset stop word table, calculating word frequency and inverse text frequency of the initial text to complete stop word filtering of the initial text; the initial text after stop word filtering is input into a pre-trained target model, a target text is obtained, and the target text is a text comprising a plurality of basic semantic words; performing part-of-speech tagging processing on the target text to generate a to-be-retrieved text meeting retrieval requirements; wherein the target model is a word segmentation model based on semantic dynamic window adjustment. The invention further provides an unstructured text processing device and equipment, a storage medium and a program product.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly to a non-structured text processing method, device, equipment, storage medium and program product. BACKGROUND

[0002] In the current financial field, with the iteration of business and technology, new technical terms and business vocabularies are constantly emerging, resulting in frequent changes in the term system, so the processing of non-structured text becomes particularly important.

[0003] Current non-structured text processing technologies mostly rely on static dictionaries and simple text processing models, which cannot adapt to rapidly changing language environments and complex semantic relationships. In addition, many existing text processing methods also have limitations in part-of-speech tagging and semantic matching, which can easily lead to information omission or misinterpretation. These problems not only affect the processing efficiency of non-structured text in the financial field, but also affect the accuracy of text retrieval. SUMMARY

[0004] In view of the above problems, the present application provides a non-structured text processing method, device, equipment, storage medium and program product.

[0005] According to a first aspect of the present application, a non-structured text processing method is provided, comprising: obtaining an initial text to be processed, the initial text being a non-structured text; performing word frequency and inverse text frequency calculation on the initial text based on a preset stop word list to complete word filtering of the initial text; inputting the initial text after stop word filtering into a pre-trained target model to obtain a target text, the target text being a text including a plurality of basic semantic words; performing part-of-speech tagging processing on the target text to generate a to-be-retrieved text meeting retrieval requirements; and wherein the target model is a word segmentation model based on semantic dynamic window adjustment.

[0006] According to an embodiment of the present application, based on the preset stop word list, the word frequency and inverse text frequency of the initial text are calculated to complete the word filtering of the initial text, comprising: constructing a stop word list and performing initial filtering on the initial text; calculating the word frequency and inverse text frequency of each word in the initial text after initial filtering, and obtaining the word frequency-inverse text frequency value of each word; based on a preset dynamic filtering threshold, filtering out the words in the initial text whose word frequency-inverse text frequency value is lower than the dynamic filtering threshold.

[0007] According to the embodiment of the present application, the part-of-speech tagging processing is performed on the target text to generate the to-be-retrieved text meeting the retrieval requirement, including: performing initial tagging on a plurality of basic semantic words in the target text based on the forward and backward features of the text; performing conditional entropy calculation and regularization processing on the basic semantic words in the target text after the initial tagging, to obtain the to-be-retrieved text meeting the retrieval requirement.

[0008] According to the embodiment of the present application, the target model is pre-trained, and the training process includes: obtaining a training set, the training set including a plurality of training texts; storing each training text according to the word length based on a preset target dictionary, to obtain a plurality of subtexts of each training text; performing semantic matching on each subtext by a dynamic window based on a preset matching algorithm, to complete the word segmentation processing of each subtext, and obtain a target text including a plurality of basic semantic words.

[0009] According to the embodiment of the present application, the training process further includes: under the condition that the target dictionary cannot meet the requirement of storing each training text according to the word length, determining the unregistered words of the training text that cannot be classified by using a preset recognition method; and adding the recognized unregistered words to the target dictionary.

[0010] According to the embodiment of the present application, the training process further includes: obtaining the target text after the initial tagging; and adjusting the size of the dynamic window based on the target text after the initial tagging.

[0011] According to the embodiment of the present application, the size of the dynamic window is adjusted, including: fixed adjustment according to the preset word length; or dynamic self-adaptive adjustment according to the context semantics and the word distribution.

[0012] The second aspect of the present application provides an unstructured text processing apparatus, including: an initial text acquisition module, configured to acquire an initial text to be processed, the initial text being an unstructured text; a filtering module, configured to calculate the word frequency and the inverse text frequency of the initial text based on a preset stop word list, to complete the word filtering of the initial text; a target text acquisition module, configured to input the initial text after the stop word filtering into a pre-trained target model, to obtain a target text, the target text being a text including a plurality of basic semantic words; and a to-be-retrieved text generation module, configured to perform part-of-speech tagging processing on the target text, to generate a to-be-retrieved text meeting the retrieval requirement; wherein the target model is a word segmentation model based on semantic dynamic window adjustment.

[0013] The third aspect of the present application provides an electronic device, including: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0014] The fourth aspect of the present application further provides a computer readable storage medium, having stored thereon a computer program or instructions, which, when executed by a processor, implement the steps of the above method.

[0015] The fifth aspect of the present application further provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implement the steps of the above method. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above content of the present application and other purposes, features and advantages will be more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0017] Figure 1 An application scenario diagram of the unstructured text processing method according to the embodiments of the present application is schematically shown;

[0018] Figure 2 A flowchart of the unstructured text processing method according to the embodiments of the present application is schematically shown;

[0019] Figure 3 A flowchart of the word filtering of the initial text according to the embodiments of the present application is schematically shown;

[0020] Figure 4 A flowchart of the target model training according to the embodiments of the present application is schematically shown;

[0021] Figure 5 A flowchart of the part-of-speech tagging processing of the target text according to the embodiments of the present application is schematically shown;

[0022] Figure 6 A structural block diagram of the unstructured text processing apparatus according to the embodiments of the present application is schematically shown; and

[0023] Figure 7 A block diagram of an electronic device suitable for implementing the unstructured text processing method according to the embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0024] Hereinafter, the embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary, and are not intended to limit the scope of the present application. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that one or more embodiments can be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques have been omitted to avoid unnecessarily obscuring the concepts of the present application.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the terms "comprises", "comprising", "includes", "including" and the like are, where used, meant to be inclusive, but not limiting in any way.

[0026] All terms used herein including technical and scientific terms have the meanings commonly understood by one of ordinary skill in the art unless otherwise specified. It should be noted that the terms used herein are to be interpreted as having a meaning that is consistent with the understanding of those terms by those skilled in the art, and should not be interpreted in an idealized or overly formal sense.

[0027] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted that the meaning of the expression is the same as "at least one of A, B, and C, etc." as understood by one of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0028] In the technical solutions of the present application, the user information (including but not limited to user personal information, user image information, user equipment information, such as location information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved are information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards, necessary security measures are taken, do not violate public order and good customs, and provide corresponding operation portal for user to choose authorization or refusal.

[0029] In the scenario of using personal information for automated decision-making, the method, device and system provided by the embodiments of the present application all provide corresponding operation portal for the user to choose to agree or refuse the automated decision-making result; if the user chooses to refuse, the expert decision-making process is entered. The expression "automated decision-making" here refers to the activity of automatically analyzing, evaluating the behavior habits, interests and hobbies, or economic, health, credit status, etc. of a person through a computer program and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by personnel who are engaged in a certain field of work, have special experience, knowledge and skills, and reach a certain professional level.

[0030] Embodiments of the present application provide a non-structured text processing method, comprising: obtaining an initial text to be processed, the initial text being a non-structured text; performing word frequency and inverse text frequency calculation on the initial text based on a preset stop word list to complete word filtering of the initial text; inputting the initial text after stop word filtering into a pre-trained target model to obtain a target text, the target text being a text including a plurality of basic semantic words; performing part-of-speech tagging processing on the target text to generate a to-be-searched text meeting search requirements; and wherein the target model is a dynamic window adjustment-based word segmentation model.

[0031] Through embodiments of the present application, the method can effectively remove stop words by combining word frequency and inverse text frequency to dynamically filter the initial text, thereby improving the accuracy of subsequent processing. In addition, the intelligent word segmentation and text processing model based on dynamic semantic window adjustment can generate high-quality to-be-searched text meeting search requirements, which not only effectively improves the processing efficiency and accuracy of non-structured text, but also greatly improves the efficiency and accuracy of information search.

[0032] Figure 1 An application scenario diagram of the non-structured text processing method according to an embodiment of the present application is schematically shown.

[0033] As shown in Figure 1 , the application scenario 100 according to this embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, and a server 105. The network 104 is a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0034] A user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0036] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0037] It should be noted that the unstructured text processing method provided in this application embodiment can generally be executed by server 105. Correspondingly, the unstructured text processing device provided in this application embodiment can generally be located in server 105. The unstructured text processing method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the unstructured text processing device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0039] The following will be based on Figure 1 The described scene, through Figures 2-5 The unstructured text processing method according to embodiments of this application will be described in detail.

[0040] Figure 2 A flowchart illustrating an unstructured text processing method according to an embodiment of this application is shown.

[0041] like Figure 2 As shown, according to an embodiment of this application, the unstructured text processing method specifically includes operations S210 to S240.

[0042] In operation S210, the initial text to be processed is obtained. The initial text is unstructured text.

[0043] In the embodiments of this application, in scenarios where related text is processed, such as text retrieval, the obtained text is mostly unstructured technical text. These texts exhibit multi-dimensional and significant features, covering a large number of specific domain terms, such as professional terms in the financial field like "data lake" and "marketing system". Because they appear infrequently in general corpora, they pose domain-specific requirements for text processing.

[0044] For example, for the field of financial banks, the obtained unstructured text to be processed contains the content of "please synchronize these data tables to the data lake".

[0045] In operation S220, the word frequency and the inverse document frequency of the initial text are calculated based on the preset stop word list, so as to complete the word filtering of the initial text.

[0046] Figure 3 The flowchart of filtering words from the initial text according to the embodiment of the application is schematically shown.

[0047] As shown in the embodiment of the application, the word frequency and the inverse document frequency of the initial text are calculated based on the preset stop word list, so as to complete the word filtering of the initial text, and the process specifically includes operation S310~operation S330. Figure 3

[0048] In operation S310, the stop word list is constructed, and the initial filtering of the initial text is performed.

[0049] Exemplarily, a mixed list composed of general Chinese stop words and bank field exclusive stop words (i.e., the stop word list) is constructed. For example, the general stop words include high-frequency words such as "of", "was", "and" which are difficult to carry effective semantics in any scene, and the bank field exclusive stop words such as "as follows", "please refer to", "see the following text". It can be understood that these stop words are derived from statistical analysis of a large amount of business text, and have the characteristics of frequent appearance and low semantic carrying degree, and all the stop words are stored in the form of an unordered set.

[0050] Further, based on the established stop word list, the initial text is initially filtered, the word list is traversed, and the stop words in the initial text are quickly matched and deleted.

[0051] In operation S320, the word frequency and the inverse document frequency of each word in the initial text after the initial filtering are calculated, and the word frequency-inverse document frequency value of each word is obtained.

[0052] In the embodiment of the application, the word frequency and the inverse document frequency of each word in the filtered initial text are calculated, and the word frequency calculation result and the inverse document frequency result are obtained. For the obtained text, when the word "data lake" appears in two texts, and the remaining words each appear in one text. Based on this, the inverse document frequency value of each word can be calculated, but the inverse document frequency value of the word "data lake" will be lower than that of the remaining words.

[0053] Further, the word frequency-inverse document frequency of each word is obtained by multiplying the word frequency and the inverse document frequency. ​

[0054] In operation S330, words with a term frequency-inverse document frequency value lower than the preset dynamic filtering threshold are filtered out from the initial text based on the preset dynamic filtering threshold.

[0055] For example, a dynamic filtering threshold is set, for example, 0.03. The term frequency-inverse document frequency value of each word is filtered, and the term frequency-inverse document frequency values of all words in the initial text are traversed. Words higher than the threshold are retained, and words lower than the threshold are filtered out. After filtering, the final vocabulary is obtained.

[0056] Through the embodiments of the present application, by combining term frequency and inverse document frequency to dynamically filter the initial text, stop words can be effectively removed, and basic semantic words useful for text semantics are retained, thereby improving the accuracy of subsequent processing.

[0057] In operation S230, the initial text after stop word filtering is input into the pre-trained target model to obtain a target text, which is a text including a plurality of basic semantic words.

[0058] In the embodiments of the present application, the text after stop word filtering is input into the pre-trained target model. The target model here can be a natural language processing model based on deep learning, more specifically, a word segmentation model based on semantic dynamic window adjustment. After processing by the target model, the model here outputs a target text including a plurality of basic semantic words according to the input semantic information. The training process of the target model is described in detail below.

[0059] Figure 4 A flowchart of training of the target model according to an embodiment of the present application is schematically shown.

[0060] As shown in FIG. 4, according to an embodiment of the present application, the target model is pre-trained, and the training process includes operations S410-S430. Figure 4 In operation S410, a training set is obtained, which includes a plurality of training texts.

[0061] In operation S420, each training text is stored according to word length based on a preset target dictionary to obtain a plurality of subtexts of each training text.

[0062] In the embodiments of the present application, a training set is obtained, which includes a plurality of training texts. These training texts are input into the constructed target model, and each training text is stored according to word length based on a preset target dictionary to obtain a plurality of subtexts of each training text.

[0063]

[0064] ​Exemplarily, a dictionary dedicated to the technical field of banks is constructed, and the terms in the dictionary are mainly derived from internal systems of banks, technical texts, and industry standards, etc. To improve the query efficiency, the dictionary structure is implemented by using a hash table to support fast query operations, and the target model stores each to-be-trained text in layers according to the word length to obtain multiple subtexts of each to-be-trained text, so as to reduce the time consumption in the matching process and improve the segmentation efficiency.

[0065] In operation S430, each subtext is subjected to semantic matching by using a dynamic window based on a preset matching algorithm, so as to complete the segmentation of each subtext and obtain a target text containing multiple basic semantic words.

[0066] Exemplarily, the matching algorithm here can be a segmentation algorithm, such as a forward maximum matching algorithm. In the process of using the forward maximum matching algorithm to perform semantic matching on the multiple subtexts of each to-be-trained text, the size of the matching window is dynamically adjusted according to the context information of the text, and longer terms, such as the professional term “public marketing system”, are preferentially matched in the matching process. In an embodiment, the adjustment manner of the window size can be fixed adjustment according to a preset word length, or dynamic adaptive adjustment according to the context semantics and the distribution of words, so as to ensure that the longest valid term can be accurately matched.

[0067] Further, based on the segmentation of each subtext, a target text containing multiple basic semantic words is input. The basic semantic word here can be understood as a basic semantic unit carrying semantics, which cannot be further segmented, such as a single character or a fixed collocation word.

[0068] In another embodiment of the present application, the training process further includes: when the target dictionary cannot satisfy the condition of storing each to-be-trained text in layers according to the word length, using a preset recognition method to determine the unregistered words of the to-be-trained text that cannot be classified; and adding the recognized unregistered words to the target dictionary.

[0069] Exemplarily, if there are unrecorded terms, that is, unregistered words, in the dictionary, the target dictionary cannot satisfy the condition of storing each to-be-trained text in layers according to the word length, that is, the to-be-trained text cannot be stored in layers. Based on this, the present embodiment proposes to use a preset recognition method to determine the unregistered words of the to-be-trained text that cannot be classified, and then add the recognized unregistered words to the target dictionary.

[0070] Specifically, an improved method combining mutual information and adjacency entropy is used to identify out-of-vocabulary (OV) words in unclassifiable training texts. First, the co-occurrence probability of word pairs is calculated to identify candidate words with high cohesion. Second, the adjacency entropy of candidate words is calculated to assess their potential to become independent terms. Then, based on a preset word frequency-adjacency entropy threshold, it is determined whether the candidate word is an OV word. Finally, the identified OV words are dynamically added to the dictionary to enrich its content and improve the accuracy of subsequent word segmentation.

[0071] Through the embodiments of this application, the word segmentation model based on semantic dynamic window adjustment makes the word segmentation process more flexible, and can adaptively adjust according to changes in context to ensure accurate semantic transmission. Furthermore, the dynamic window processing method using a preset target dictionary and matching algorithm can better adapt to different types of text, improve the model's training effect, and provide an effective mechanism for identifying and adding out-of-vocabulary words, enhancing the model's flexibility and adaptability.

[0072] In operation S240, part-of-speech tagging is performed on the target text to generate the search text that meets the search requirements.

[0073] Figure 5 The flowchart illustrating part-of-speech tagging of target text according to an embodiment of this application is shown in the illustration.

[0074] like Figure 5 As shown, according to an embodiment of this application, the target text is subjected to part-of-speech tagging to generate a searchable text that meets the search requirements. This specific process includes operations S510 to S520.

[0075] In the S510 operation, based on the forward and backward features of the text, multiple basic semantic words in the target text are initially labeled.

[0076] In the embodiments of this application, an initial category label is automatically assigned to each basic unit (character or word) in the text by utilizing the context information of a target word in the text.

[0077] For example, a predefined annotation set and feature template are used to input the target text, which includes multiple basic semantic words, output by the target model into a pre-trained statistical machine learning model, and to predict a label that conforms to the context semantics for each basic semantic word.

[0078] In operation S520, the basic semantic words in the target text after initial annotation are subjected to conditional entropy calculation and regularization processing to obtain the text to be retrieved that meets the retrieval requirements.

[0079] In the embodiments of the present application, the results of the initial labeling are quantitatively evaluated and cleaned, the low-information or irrelevant words are filtered out, and the target text representation is reshaped to be more "retrieval-requirement-compliant" (i.e., to improve the accuracy and recall rate of retrieval).

[0080] Specifically, the conditional entropy of each basic semantic word is calculated, and then the entropy value is regularized based on the entropy value to filter out low-information words and weight high-information words, and finally a weighted and pure keyword set is generated, that is, the output of the text to be retrieved.

[0081] In the embodiments of the present application, the above-mentioned initial labeled basic semantic words can also be fed back to the target model for training of the target model. It can be understood here that the results of subsequent labeling are used to correct errors that may occur in the above-mentioned S430 word segmentation process. For example, in the expression "table synchronization to data lake", the word "synchronization" can be used as a noun or a verb, and there is a word segmentation ambiguity. By combining the part-of-speech labeling result (determined as a verb), the sentence can be more accurately segmented, the ambiguity can be eliminated, and the accuracy of word segmentation can be improved. For another example, the dynamic window can also be adjusted through the results of subsequent labeling to adjust the training process. For example, after obtaining the target text after initial labeling, the size of the dynamic window is adjusted based on the target text after initial labeling to correct errors generated by word segmentation and improve the accuracy of word segmentation.

[0082] Through the embodiments of the present application, the initial labeling method combining forward and backward features, combined with conditional entropy and regularization processing, can generate high-quality text to be retrieved that meets the retrieval requirements, and greatly improves the efficiency and accuracy of information retrieval.

[0083] Based on the above-mentioned unstructured text processing method, the present application also provides an unstructured text processing device. The following will combine Figure 6 the device will be described in detail.

[0084] Figure 6 The structure block diagram of the unstructured text processing device according to the embodiments of the present application is schematically shown.

[0085] As Figure 6 shown, the unstructured text processing device 600 of the embodiments includes an initial text acquisition module 610, a filtering module 620, a target text acquisition module 630, and a text to be retrieved generation module 640.

[0086] The initial text acquisition module 610 is configured to acquire the initial text to be processed, and the initial text is an unstructured text. In an embodiment, the initial text acquisition module 610 can be configured to perform the operation S210 described above, and details are not repeated here.

[0087] The filtering module 620 is configured to perform word frequency and inverse document frequency calculation on the initial text based on the preset stop word list, so as to complete word filtering of the initial text. In an embodiment, the filtering module 620 can be configured to perform the operation S220 described above, and details are not repeated here.

[0088] The target text acquisition module 630 is configured to input the initial text after the stop word filtering into a pre-trained target model, to obtain a target text. The target text is a text including a plurality of basic semantic words. In an embodiment, the target text acquisition module 630 can be configured to perform the operation S230 described above, and details are not repeated here.

[0089] The text to be searched generation module 640 is configured to perform part-of-speech tagging processing on the target text, to generate a text to be searched meeting the search requirement. In an embodiment, the text to be searched generation module 640 can be configured to perform the operation S240 described above, and details are not repeated here.

[0090] In an embodiment of the present application, the target model is a word segmentation model based on semantic dynamic window adjustment.

[0091] In an embodiment of the present application, the filtering module 620 includes an initial filtering sub-module, a word frequency calculation sub-module, and a word filtering sub-module.

[0092] The initial filtering sub-module is configured to construct a stop word list and perform initial filtering on the initial text. In an embodiment, the initial filtering sub-module can be configured to perform the operation S310 described above, and details are not repeated here.

[0093] The word frequency calculation sub-module is configured to calculate the word frequency and inverse document frequency of each word in the initial text after the initial filtering, and obtain a word frequency-inverse document frequency value of each word. In an embodiment, the word frequency calculation sub-module can be configured to perform the operation S320 described above, and details are not repeated here.

[0094] The word filtering sub-module is configured to filter out a word in the initial text whose word frequency-inverse document frequency value is lower than a preset dynamic filtering threshold. In an embodiment, the word filtering sub-module can be configured to perform the operation S330 described above, and details are not repeated here.

[0095] In an embodiment of the present application, the target text acquisition module 630 includes a first training sub-module, a second training sub-module, and a third training sub-module.

[0096] The first training submodule is configured to obtain a training set, and the training set comprises a plurality of training texts; based on a preset target dictionary, each training text is classified and stored according to a word length, and a plurality of subtexts of each training text are obtained; based on a preset matching algorithm, each subtext is semantically matched through a dynamic window, so as to complete the word segmentation of each subtext, and a target text comprising a plurality of basic semantic words is obtained. In an embodiment, the first training submodule can be configured to perform the operation S410 to the operation S430 described above, and details are not described herein again.

[0097] The second training submodule is configured to, when the target dictionary cannot satisfy the condition that each training text is classified and stored according to the word length, determine an out-of-vocabulary word of a training text that cannot be classified by using a preset identification method; and add the identified out-of-vocabulary word to the target dictionary.

[0098] The third training submodule is configured to obtain an initial annotated target text; and based on the initial annotated target text, the size of the dynamic window is adjusted. The third training submodule further comprises a dynamic adjustment unit, which is configured to perform fixed adjustment according to a preset word length, or perform dynamic self-adaptive adjustment according to context semantics and word distribution.

[0099] According to embodiments of the present application, any one of the initial text acquisition module 610, the filtering module 620, the target text acquisition module 630 and the to-be-retrieved text generation module 640 can be combined in one module, or any one of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of other modules, and implemented in one module. According to embodiments of the present application, at least one of the initial text acquisition module 610, the filtering module 620, the target text acquisition module 630 and the to-be-retrieved text generation module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system in package, an application specific integrated circuit (ASIC), or any other reasonable manner of integrating or packaging a circuit, etc. hardware or firmware, or implemented in any one of software, hardware and firmware or in a proper combination of any of them. Alternatively, at least one of the initial text acquisition module 610, the filtering module 620, the target text acquisition module 630 and the to-be-retrieved text generation module 640 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0100] Figure 7 A block diagram of an electronic device suitable for implementing the unstructured text processing method according to embodiments of the present application is schematically shown.

[0101] As shown in FIG. 7, the electronic device 700 according to an embodiment of the present application includes a processor 701 which can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 702 or a program loaded into a random access memory (RAM) 703 from a storage section 708. The processor 701 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and the like. The processor 701 can also include an on-board memory for cache use. The processor 701 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present application. Figure 7 In the RAM 703, various programs and data required for the operation of the electronic device 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations of the method processes according to embodiments of the present application by executing the programs in the ROM 702 and / or the RAM 703. Note that the programs can also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 can also perform various operations of the method processes according to embodiments of the present application by executing the programs stored in the one or more memories.

[0102] According to embodiments of the present application, the electronic device 700 can further include an input / output (I / O) interface 705 which is also connected to the bus 704. The electronic device 700 can further include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage section 708 including a hard disk, and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, and the like. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as necessary. A removable medium 711 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 710 as necessary, so that a computer program read out therefrom is installed into the storage section 708 as necessary.

[0103]

[0104] ​The application further provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist independently without being assembled into the device / apparatus / system. The computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0105] According to the embodiments of the application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In this application, a computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to the embodiments of the application, the computer readable storage medium can include the ROM 702 and / or the RAM 703 described above, and / or one or more memory other than the ROM 702 and the RAM 703.

[0106] The embodiments of the application also include a computer program product, which includes a computer program containing program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the unstructured text processing method provided by the embodiments of the application.

[0107] The above functions defined in the system / apparatus of the embodiments of the application are performed when the computer program is executed by the processor 701. According to the embodiments of the application, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0108] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium, and be downloaded and installed through the communication part 709, and / or installed from the detachable medium 711. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0109] In such embodiments, the computer program can be downloaded and installed from the network via the communication section 709, and / or installed from the removable media 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiments of the present application are executed. According to the embodiments of the present application, the system, device, apparatus, module, unit, and the like described above can be realized by the computer program modules.

[0110] According to the embodiments of the present application, the program code for executing the computer program provided by the embodiments of the present application can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language, or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0111] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0112] Those skilled in the art can understand that the features described in various embodiments of the present application can be combined and / or integrated in various combinations and / or integrations, even if such combinations or integrations are not explicitly described in the present application. In particular, the features described in various embodiments of the present application can be combined and / or integrated in various combinations and / or integrations without departing from the spirit and teachings of the present application. All such combinations and / or integrations fall within the scope of the present application.

Claims

1. A method for processing unstructured text, characterized in that, The method includes: Obtain the initial text to be processed, wherein the initial text is unstructured text; Based on a preset stop word list, the initial text is subjected to word frequency and inverse text frequency calculations to complete word filtering of the initial text; The initial text after stop word filtering is input into the pre-trained target model to obtain the target text, which is a text that includes multiple basic semantic words; The target text is subjected to part-of-speech tagging to generate searchable text that meets the search requirements; The target model is a word segmentation model based on semantic dynamic window adjustment.

2. The unstructured text processing method according to claim 1, characterized in that, The process of calculating word frequency and inverse text frequency of the initial text based on a preset stop word list to filter the words in the initial text includes: Construct a stop word list and perform initial filtering on the initial text; Calculate the term frequency and inverse text frequency of each word in the initial text after initial filtering, and obtain the term frequency-inverse text frequency value of each word; Based on a preset dynamic filtering threshold, words in the initial text whose word frequency-inverse text frequency value is lower than the dynamic filtering threshold are filtered out.

3. The unstructured text processing method according to claim 1, characterized in that, The step of performing part-of-speech tagging on the target text to generate searchable text that meets the search requirements includes: Based on the forward and backward features of the text, multiple basic semantic words in the target text are initially labeled. Conditional entropy calculation and regularization are performed on the basic semantic words in the target text after initial annotation to obtain the search text that meets the search requirements.

4. The unstructured text processing method according to claim 1, characterized in that, The target model is pre-trained, and the training process includes: Obtain a training set, which includes multiple texts to be trained; Based on the preset target dictionary, each text to be trained is classified and stored according to word length, resulting in multiple sub-texts for each text to be trained. Based on a preset matching algorithm, each sub-text is semantically matched through a dynamic window to complete the word segmentation of each sub-text and obtain the target text containing multiple basic semantic words.

5. The unstructured text processing method according to claim 4, characterized in that, The training process also includes: If the target dictionary cannot classify and store each training text according to word length, a preset recognition method is used to identify out-of-vocabulary words in the training texts that cannot be classified. The identified out-of-vocabulary words are added to the target dictionary.

6. The unstructured text processing method according to claim 3 or 4, characterized in that, The training process also includes: Obtain the target text after initial annotation; The size of the dynamic window is adjusted based on the target text after the initial annotation.

7. The unstructured text processing method according to claim 6, characterized in that, Adjusting the size of the dynamic window includes: Adjustments are made based on a preset word length; or The system dynamically and adaptively adjusts based on contextual semantics and vocabulary distribution.

8. An unstructured text processing device, characterized in that, The device includes: The initial text acquisition module is used to acquire the initial text to be processed, wherein the initial text is unstructured text; The filtering module is used to calculate the word frequency and inverse text frequency of the initial text based on a preset stop word list, so as to complete the word filtering of the initial text; The target text acquisition module is used to input the initial text after stop word filtering into the pre-trained target model to obtain the target text, which is a text including multiple basic semantic words; The target text generation module is used to perform part-of-speech tagging on the target text to generate a target text that meets the retrieval requirements; The target model is a word segmentation model based on semantic dynamic window adjustment.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images