Text processing method and device

Abstract

The embodiment of the present application provides a text processing method and device, including: obtaining a term of at least one domain category; obtaining the associated content of the term; and determining the degree of match between the term and the domain category to which the term belongs based on the associated content of the term and the identification data set of the domain category to which the term belongs. In the present application, the associated content of the term can greatly expand the dimension of the term, and the identification data set can also greatly expand the dimension of the corresponding domain category. In the process of determining the degree of match using the associated content of the term and the identification data set of the domain category to which the term belongs, the associated content and the identification data set can provide richer semantic information, thereby improving the accuracy of quality management of term data. In addition, the entire quality management strategy can be implemented through automated information mining, information analysis, and information comparison, thereby greatly reducing manual participation, improving generation efficiency, and reducing production costs.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text processing method and device. Background Art

[0002] Domain-specific terminology data has broad application prospects. Terminology data can be widely used in technical fields such as e-commerce, social networking, and machine translation, thereby meeting the needs of these technical fields for professionalism and standardization of text data.

[0003] At present, manual or automated term mining links can be used to mine term data in various fields and categories. With the continuous development of the technical fields to which term data are applied, the requirements for the quality of term data are also constantly increasing. However, the above-mentioned term data mining method has the problem of uneven quality of the generated term data. The existing solution is to use manual picking to perform quality analysis and screening on the mined term data, so as to achieve the purpose of controlling the quality of term data.

[0004] However, in the current solution, the quality management of terminology data is performed manually, which results in high labor costs and low efficiency, and also leads to low accuracy. Summary of the invention

[0005] The embodiment of the present application provides a text processing method to achieve quality management of terminology data in a low-cost manner under an automated terminology data quality verification strategy, thereby improving management efficiency and accuracy.

[0006] Correspondingly, the embodiment of the present application also provides a text processing device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the present application discloses a text processing method, which includes:

[0008] Get at least one term of the domain category;

[0009] Obtaining content associated with the term;

[0010] The degree of matching between the term and the field category to which the term belongs is determined based on the associated content of the term and the identification data set of the field category to which the term belongs.

[0011] The present application also discloses a text processing device, which includes:

[0012] A first acquisition module, used to acquire terms of at least one field category;

[0013] A second acquisition module is used to acquire the associated content of the term;

[0014] The verification module is used to determine the matching degree between the term and the field category to which the term belongs based on the associated content of the term and the identification data set of the field category to which the term belongs.

[0015] An embodiment of the present application further discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes one or more methods described in the embodiments of the present application.

[0016] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes one or more methods described in the embodiments of the present application.

[0017] Compared with the prior art, the embodiments of the present application have the following advantages:

[0018] In the embodiments of the present application, the associated content of a term can greatly expand the dimension of the term, so that the definition scope of the term is not limited to one term, but the definition scope of the term is greatly expanded by the associated content with a wider breadth. Similarly, the identification data set can also greatly expand the dimension of the corresponding field category. In the process of determining the degree of match using the associated content of the term and the identification data set of the field category to which the term belongs, the associated content and the identification data set can provide richer semantic information. A large amount of semantic information helps to improve the accuracy of calculating the degree of match, thereby improving the accuracy of quality management of term data. In addition, the entire quality management strategy can be implemented by automated information mining, information analysis, and information comparison, thereby greatly reducing manual participation, improving generation efficiency, and reducing production costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a system architecture diagram of a text processing method according to an embodiment of the present application;

[0020] Figure 2 is a system architecture diagram of another text processing method according to an embodiment of the present application;

[0021] Figure 3 This is a text processing system architecture diagram for a machine translation scenario in this application;

[0022] Figure 4 This is a text processing system architecture diagram for another machine translation scenario of the present application;

[0023] Figure 5 This is a text processing system architecture diagram for another machine translation scenario of the present application;

[0024] Figure 6 This is a diagram of the architecture of a text processing system in the medical field of the present application;

[0025] Figure 7 is a flowchart of the steps of a text processing method of the present application;

[0026] Figure 8 It is a flowchart of the specific steps of a text processing method of the present application;

[0027] Fig. 9 It is a flowchart of a matching degree solving process of the present application;

[0028] Fig.10 is a structural block diagram of an embodiment of a text processing device of the present application;

[0029] Fig.11 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0031] Reference Figure 1 , which shows a system architecture diagram of a text processing method provided in an embodiment of the present application, including: a terminology data mining server and a terminology data verification server.

[0032] Among them, the terminology data mining server can perform terminology data mining based on data sources such as lexicons, websites, and databases to obtain terms in multiple field categories. Terms belonging to a certain field category can be used as proprietary identifiers in that field. Terms can use text as a carrier to express or limit the conventional symbols of professional concepts. In addition, terms can also use other forms of carriers, such as symbols.

[0033] Specifically, the data source as the source of term data can be in various forms. In one implementation, the data source can be an encyclopedia knowledge website. By using a certain field category as a search term, the content of the term corresponding to the field category is searched, and the term content is mined to obtain the terms under the field category. For example, in the medical field, the "medical" term is searched on the encyclopedia knowledge website to obtain the web page of the term content, which has descriptions such as "medical is a Chinese word with two meanings of treatment and disease treatment", "health is a common pursuit of people, but disease also accompanies life all the way." Through term mining, "treatment", "disease", "health" and "treatment" can be extracted as terms in the medical field. In another implementation, the data source can be a search engine. By retrieving web page data associated with the field category in the search engine, it is possible to mine terms from the associated web page data. For example, tools such as common crawl are used to retrieve web page data associated with the field category in the search engine, and mine terms in the associated web page data.

[0034] Furthermore, the data source may be a thesaurus or a database. The thesaurus may include a dictionary thesaurus, such as an industrial dictionary, a medical dictionary, etc., or may include a network thesaurus, an input method thesaurus, etc. The database may include a translation software database, a commodity database, etc. Since the thesaurus and the database already contain properties such as the special effects and categories of words, the terms in various field categories can be mined through simple screening.

[0035] Based on the raw data obtained from the data source, the term data mining server can use a variety of mining methods to obtain terms of various fields and categories from the raw data. For example, when mining the terminology of the raw text data, the term frequency, degree of formation, and context of the words in the raw text data can be used in combination with vector space models, term frequency-inverse document frequency (TFIDF, erm Frequency-Inverse Document Frequency), Bayesian reasoning and other methods to achieve term mining.

[0036] The terminology data verification server can obtain the terms of various field categories mined by the terminology data mining server from the terminology data mining server, and further verify the matching degree of each term with the field category to which it belongs. The greater the matching degree, the higher the quality of the term. The matching degree quantifies the quality of the term, thereby achieving control over the quality of the term, such as removing terms with a small matching degree from the field category.

[0037] Further, the application scenarios of the terminology verification results generated by the terminology data verification server are described, refer to Figure 2, which shows a system architecture diagram of another text processing method provided in an embodiment of the present application, including: a terminology data mining server, a terminology data verification server and a terminology data application server.

[0038] Among them, the terminology data application server can actually apply the terminology data and establish a terminology database locally to provide terminology support for the services it provides.

[0039] Reference Figure 3 , which shows a text processing system architecture diagram for a machine translation scenario provided in an embodiment of the present application.

[0040] In the embodiment of the present application, the machine translation scenario relies on a large number of high-quality professional terms, and the industries involved in machine translation are very extensive, such as: medical, education, automobile, etc. Different industries have different focuses on the customization requirements for machine translation. Building a domain-specific machine translation model requires not only high-quality domain bilingual data, but also terminology data of domain classes that are indispensable and important for building a high-quality domain machine translation model. Therefore, for the machine translation scenario, the terminology data application server can be a translation service implementation end, and the terminology data application server can be connected to the terminology library of each field. The terminology library of each field can be set locally at the terminology data application server, or it can be set in the cloud. The terminology data application server can upload the terms in the terminology library of each field to the terminology data verification server, and after verification by the terminology data verification server, the terminology matching degree is sent to the terminology data application server, and the terminology data application server can correct, screen, and reclassify the terms in the terminology library of each field according to the term matching degree. Afterwards, the terminology data application server can receive the text to be translated uploaded by the client, and determine one or more target domain categories associated with the text to be translated based on the semantic recognition of the text to be translated, and select terms in the target domain category for use during the translation process, or when the translation is initially completed, select terms in the target domain category to replace non-professional words in the preliminary translation text, so as to achieve the purpose of producing a higher quality translation text. Finally, the terminology data application server can send the translated text to the corresponding user's client to provide the user with high-quality translation services.

[0041] Among them, the user can be an individual user, the user can also be an agency for various matters, such as a patent agency, the user can also be a patent management department in various regions, the user can also be various corporate users, such as law firms, translation companies, foreign trade companies, etc., the user can also be various public institutions, related units, etc., and the embodiments of the present application do not limit them.

[0042] In the embodiment of the present application, for individual users, there may be the following three interaction schemes:

[0043] (1) Individual users can upload texts to be translated to the terminology data application server through their personal clients. The terminology data application server translates the texts to be translated based on high-quality terminology, thereby generating high-quality translated texts and providing them to the personal clients, so that the individual users can study the translated texts on their personal clients.

[0044] (2) Individual users can upload the text to be translated to the terminology data application server through the personal client. The terminology data application server uses the original translation algorithm to translate the text to be translated, and then selects high-quality terms to replace non-professional words in the translation results, thereby generating high-quality translation texts and providing them to the personal client and the translation company client, so that individual users can study the translation texts on the personal client. Enterprise users can obtain the translated texts through the translation company client, thereby reducing the translation costs of the enterprise. The text to be translated and the translation results can be in two different languages.

[0045] (3) The personal client uploads the translated text to the terminology data application server. The terminology data application server calibrates the translated text based on high-quality terminology and selects high-quality terminology to replace non-professional words in the translated text during the calibration process, thereby generating high-quality translated text and providing it to the personal client so that the personal user can study the translated text on the personal client.

[0046] It should be noted that the above-mentioned individual users can be replaced by the above-mentioned various corporate users, such as the above-mentioned law firms, translation companies, foreign trade companies, etc., so that the corporate user's client can perform similar interactions with the above-mentioned terminology data application server, which can reduce the translation cost of the corporate user and improve translation efficiency.

[0047] In addition, in (1), the terminology data application server can also establish a corresponding relationship between the professional terms in the original language of the text to be translated and the high-quality terms used for translation and send them to the personal client for display. During the display process, the translated text and the text to be translated can be displayed in correspondence; wherein, the association relationship between the high-quality term and the professional term in the original language is displayed, for example, in the text to be translated and the translated text, the high-quality term and the professional term in the original language are connected with a connecting line to reflect the corresponding relationship between the two. Of course, brackets can also be added after the high-quality term in the translated text, and the professional term in the original language is displayed in the brackets. This embodiment of the present application does not limit it.

[0048] In (2) and (3), the terminology data application server can also establish a correspondence between the non-professional term and the high-quality term used to replace the non-professional term and send it to the personal client for display. During the display process, the non-professional term and the high-quality term can be displayed in correspondence, and the two terms can be highlighted, such as bold display, highlight display, etc., so that the personal user of the personal client can perceive the conversion between the original language and the translated language, thereby achieving the purpose of learning the new language.

[0049] In actual applications, an input interface can be displayed in the above-mentioned client. After the user enters a word or a sentence in the client, the word or sentence is uploaded to the terminology data application server. The terminology data application server returns the above-mentioned processing result to the client. The client can display the above-mentioned processing result in the result display interface to achieve the effect of learning while translating.

[0050] Further, see Figure 4 , which shows a text processing system architecture diagram of another machine translation scenario provided by an embodiment of the present application, which can be specifically applied to the patent field. In this field, there is a demand for language reflection of patent documents. The terminology data application server can be a patent translation server, which is connected to a patent terminology library. The patent terminology library can be set locally on the patent translation server or in the cloud. The patent translation server can upload patent terms in the patent terminology library to the terminology data verification server, and after verification by the terminology data verification server, the terminology matching degree is sent to the patent translation server. The patent translation server can receive the patent document to be translated uploaded by the patent upload system according to the terminology matching degree, and determine one or more target field categories associated with the patent document to be translated according to the semantic recognition of the patent document to be translated, and select terms in the target field category for use during the translation process, or select terms in the target field category to replace non-professional words in the initially translated patent document when the translation is initially completed, so as to achieve the purpose of producing a higher quality translated patent document.

[0051] After the translation is completed, the patent translation service can send the translated high-quality patent documents to the client of the corresponding agent's agency, thereby saving the agent's translation costs. In addition, the patent translation service can also send the translated high-quality patent documents to the client of the patent review unit of the corresponding reviewer, thereby saving the reviewer the difficulty of reviewing the patent documents.

[0052] Further, see Figure 5, which shows a text processing system architecture diagram of another machine translation scenario provided by the embodiment of the present application. In the embodiment of the present application, the machine translation scenario relies on a large number of high-quality professional terms, and the industries involved in machine translation are very wide, such as: medical, education, automobile, etc. Enterprises in different industries have different focuses on the customization requirements of machine translation. Building a domain-specific machine translation model requires not only high-quality domain bilingual data, but also domain-class terminology data that plays an indispensable and important role in building a high-quality domain machine translation model.

[0053] Assuming that the enterprise corresponding to the enterprise client is an enterprise in the automotive industry, for the machine translation scenario, the terminology data application server can be the translation service implementation end, the terminology data application server can be connected to the automotive terminology library, and the automotive terminology data application server can upload the automotive terminology to the terminology data verification server. After verification by the terminology data verification server, the terminology matching degree is sent to the terminology data application server. The terminology data application server can correct, filter, reclassify, etc. the automotive terminology in the automotive terminology library according to the term matching degree. Afterwards, the terminology data application server can receive the text to be translated in the automotive field uploaded by the enterprise client, and determine the automotive field category associated with the text to be translated based on the semantic recognition of the text to be translated, and select automotive terminology in the automotive field category for use during the translation process, or select automotive terminology in the automotive field category to replace non-professional words in the preliminary translation text when the translation is initially completed, so as to achieve the purpose of producing a higher quality translation text. Finally, the terminology data application server can send the translated text to the corresponding enterprise user's enterprise client to provide the enterprise user with high-quality translation services corresponding to its field.

[0054] Furthermore, the various application fields of the terminology data application server are described in detail:

[0055] For example, refer to Figure 6, which shows an architecture diagram of a text processing system in the medical field provided by an embodiment of the present application. For the medical field, the terminology data application server can be a medical record management server, which is connected to a medical terminology library. The medical terminology library can be set locally on the medical record management server or in the cloud. The medical record management server can upload the medical terms in the medical terminology library to the terminology data verification server, and after verification by the terminology data verification server, the terminology matching degree is sent to the medical record management server. The medical record management server can correct, screen, and reclassify the terms in the medical terminology library according to the terminology matching degree. After that, the medical record management server can receive the medical record files uploaded by the doctors of the hospital through the terminal, and based on the updated medical terminology library, select corresponding high-quality terms to replace the non-standard words in the medical record files to achieve the standardization of the medical record files, and store the corrected medical record files locally, or send them to the corresponding patient's client to provide the patient with high-quality medical record files.

[0056] In the field of logistics, the terminology data application server can be a logistics management server. The logistics management server can manage the logistics documents generated in each logistics link, and select corresponding terms based on the logistics terminology library to replace non-standard words in the logistics documents, or select corresponding terms as the header content when establishing logistics documents.

[0057] For the automotive field, the terminology data application server can be an automotive forum server, which can provide a browsable automotive forum website and select corresponding terms based on the automotive terminology library to achieve website establishment and maintenance during the establishment and maintenance of the automotive forum website.

[0058] The terminology data verification service can provide the terminology data application service with the degree of match between the term and the field category to which it belongs for each field category. The terminology data application service can clean the terms in the local terminology library based on the verification results of the terms, such as removing terms with a small degree of match from the terminology library, or sorting the terms by importance according to the degree of term match, so that the services based on the terms can select terms according to the order when selecting terms for use. For example, when the medical record management service replaces a non-standard word in a medical record file, if there are multiple terms associated with the non-standard word, the term with the greatest degree of match can be selected for replacement.

[0059] In an embodiment of the present application, the terminology data verification server can pre-build a corresponding identification data set for each domain category, and based on each term, obtain the corresponding associated content, and determine the degree of match between the term and the domain category to which the term belongs based on the associated content of the term and the identification data set of the domain category to which the term belongs.

[0060] Specifically, the data in the recognition data set is used to accurately reflect the characteristics of the domain category. In one implementation, the recognition data set may include reference structured terms or reference text paragraphs pre-defined for the domain category. These defined texts can accurately reflect the characteristics of the domain category, wherein the reference structured terms can be recognized proprietary terms or high-quality terms within the domain category, and the reference text paragraph can also be called seed text, which is a text paragraph with a recognized high confidence level for accurately describing the domain.

[0061] For example, for the "medical" field category, reference structured terms may include: "treatment", "disease", "health", "treatment" and other terms, and reference text paragraphs may include: text paragraphs that provide a more authoritative description and definition of "medical", etc.

[0062] When the term is in text form, the associated content of the term may include related words of the term, such as synonyms, antonyms, aliases, Chinese / English names, etc., and may also include text paragraphs that further explain the term.

[0063] For example, for the term "coronary heart disease" in the medical field, its related content may include associated words, such as the alias "coronary atherosclerotic heart disease", the English name "coronary atherosclerotic heart disease", related terms such as "medical treatment", "medical treatment", etc., and may also include related text paragraphs: "The diagnosis of coronary heart disease mainly relies on typical clinical symptoms, combined with auxiliary examinations to find evidence of myocardial ischemia or coronary artery obstruction."

[0064] In one implementation, if the term, the term's associated content, and the identification data set are in text form, in the process of determining the degree of match between the term and the field category to which the term belongs, the first text feature of the term's associated content and the second text feature of the data in the identification data set of the field category to which the term belongs can be extracted. By analyzing the degree of association between the first text feature and the second text feature, the degree of association can be ultimately converted into the degree of match between the term and the field category to which the term belongs.

[0065] Of course, the method for calculating the degree of match between a term and the category of the field to which the term belongs in the embodiment of the present application is not limited to the comparison of text features. For example, the degree of match between a term and the category of the field to which the term belongs can also be determined based on the frequency by simply determining the frequency of occurrence of a reference structured term in the associated content of the term in the recognition data set.

[0066] For example, for the term "coronary heart disease" in the medical field category, the associated content of the term includes: associated words, such as "consultation", "medical treatment", "medicine", etc., and the associated text paragraph "The diagnosis of coronary heart disease mainly relies on typical clinical symptoms, combined with auxiliary examinations to find evidence of myocardial ischemia or coronary artery obstruction." The identification data set of the medical field category includes: reference structured terms, such as "treatment", "disease", "health", "treatment", and the reference text paragraph "The medical and health system has a clear positioning. China's medical and health has created a series of glories and has made great achievements in various aspects such as medical examinations and clinical diagnosis."

[0067] By extracting the vector features of the associated words of the associated content of the term and the reference structured terms of the identification data set of the medical field category, and calculating the vector distance between the two according to the vector features, a first score converted from the vector distance can be obtained. By extracting the vector features of the text paragraphs in the associated content of the term and the reference text paragraphs of the identification data set of the medical field category, and calculating the vector distance between the two according to the vector features, a second score converted from the vector distance can be obtained. Different weight values ​​can be set for the feature comparison results of the associated words and the reference structured terms, and the feature comparison results of the associated text paragraphs and the reference text paragraphs. After the first score and the second score are weighted and summed, the final score is used as the matching degree between the term "coronary heart disease" and the medical field category. It can be seen from the above example that since the similarity and correlation between the associated words of the term and the reference structured terms of the identification data set are large, and there is a large correlation between the associated text paragraphs of the term and the reference text paragraphs of the identification data set, the first score and the second score will be relatively high, so that the matching degree between the term "coronary heart disease" and the medical field category is high, and the term "coronary heart disease" in the medical field category has a high quality.

[0068] In the embodiments of this application, refer to Figure 1 In one implementation, the associated content acquisition module of the term data verification server can use the term as a search term, search for the corresponding term page on the knowledge website, and use part or all of the content in the corresponding term page as the associated content of the term.

[0069] For example, for the term "coronary heart disease" in the medical field, you can search for the entry page corresponding to the entry "coronary heart disease" in encyclopedia knowledge sites. The entry page may include the above-mentioned aliases, English names, text paragraphs used to describe the entry and other related information.

[0070] Reference Figure 1In another implementation, the associated content acquisition module of the term data verification server can use the term as a query word to query the associated web pages in the search engine. The corresponding query results can include multiple web links associated with the term, and the multiple web links are sorted in descending order according to the degree of relevance between the content and the term. The embodiment of the present application can filter the query results and use part or all of the content in the web pages corresponding to the web links remaining after filtering as the associated content of the term. For example, when filtering multiple web links, the web page corresponding to the web link with the highest degree of relevance can be selected, and the remaining web links can be filtered, or the web pages corresponding to several web links with the highest degree of relevance can be selected, and the remaining web links can be filtered.

[0071] It should be noted that the sources for obtaining the related content in the embodiments of the present application are not limited to knowledge websites and search engines, but may also include other sources such as lexicons and databases, which are not limited in the embodiments of the present application.

[0072] It should be noted that in order to ensure a high accuracy in solving the matching degree between a term and the field category to which the term belongs, the data volume of the term's associated content and the data volume of the field category identification data set need to be high. For example, the source web page data of the term's associated content needs to be greater than 50 web pages, or the text data volume in the entry page of the term's associated content needs to be greater than 1,000 words; the number of reference structured terms in the identification data set needs to be greater than 10,000, and the number of words in the reference text paragraph needs to be greater than 100,000 words. Among them, each data volume threshold can be set according to actual conditions, and the embodiments of the present application do not limit this. In addition, the content of the associated text paragraph and the reference text paragraph can be sentences or paragraphs.

[0073] In an embodiment of the present application, the associated content of a term can greatly expand the dimension of the term, so that the definition scope of the term is not limited to one term, but the definition scope of the term is greatly expanded by the associated content with a wider breadth. Similarly, the identification data set can also greatly expand the dimension of the corresponding field category. In the process of determining the degree of match using the associated content of the term and the identification data set of the field category to which the term belongs, the associated content and the identification data set can provide richer semantic information. A large amount of semantic information helps to improve the accuracy of calculating the degree of match, thereby improving the accuracy of quality management of term data. In addition, the entire quality management strategy can be implemented by automated information mining, information analysis, and information comparison, thereby greatly reducing manual participation, improving generation efficiency, and reducing production costs.

[0074] Reference Figure 7 , the present application embodiment provides a flowchart of the steps of a text processing method, including:

[0075] Step 101: Obtain terms of at least one field category.

[0076] In the embodiments of the present application, terms belonging to a certain field category can be used as proprietary identifiers in the field. Terms can use text as a carrier to express or define the conventional symbols of professional concepts. In addition, terms can also use other forms of carriers, such as symbols.

[0077] Specifically, refer to Figure 1 There are many forms of data sources as the source of terminology data. In one implementation, the data source can be an encyclopedia knowledge website. By using a certain field category as a search term, the content of the term corresponding to the field category is searched, and term mining is performed on the term content to obtain the terms under the field category. In another implementation, the data source can be a search engine. By retrieving web page data associated with the field category in the search engine, term mining can be achieved from the associated web page data.

[0078] Furthermore, the data source may be a thesaurus or a database. The thesaurus may include a dictionary thesaurus, such as an industrial dictionary, a medical dictionary, etc., or may include a network thesaurus, an input method thesaurus, etc. The database may include a translation software database, a commodity database, etc. Since the thesaurus and the database already contain properties such as the special effects and categories of words, the terms in various field categories can be mined through simple screening.

[0079] Step 102: Obtain the associated content of the term.

[0080] In an embodiment of the present application, the associated content of a term can greatly expand the dimension of the term, so that the definition scope of the term is not limited to one term, but the definition scope of the term is greatly expanded by the broader associated content. When the term is in text form, the associated content of the term may include the associated words of the term, such as synonyms, antonyms, aliases, Chinese / English names, etc., and may also include text paragraphs that further explain the term.

[0081] For example, for the term "coronary heart disease" in the medical field, its related content may include associated words, such as the alias "coronary atherosclerotic heart disease", the English name "coronary atherosclerotic heart disease", related terms such as "medical treatment", "medical treatment", etc., and may also include related text paragraphs: "The diagnosis of coronary heart disease mainly relies on typical clinical symptoms, combined with auxiliary examinations to find evidence of myocardial ischemia or coronary artery obstruction."

[0082] Reference Figure 1In one implementation, the associated content acquisition module of the term data verification server can use the term as a search term, search for the corresponding term page on the knowledge website, and use part or all of the content in the corresponding term page as the associated content of the term.

[0083] In another implementation, the associated content acquisition module of the term data verification server can use the term as a query word to query related web pages in the search engine. The corresponding query results may include multiple web page links associated with the term, and the multiple web page links are sorted in descending order according to the degree of relevance between the content and the term. The embodiment of the present application can filter the query results and use part or all of the content in the web page corresponding to the remaining web links after filtering as the associated content of the term.

[0084] Step 103: Determine the degree of match between the term and the field category to which the term belongs based on the associated content of the term and the identification data set of the field category to which the term belongs.

[0085] In an embodiment of the present application, in one implementation method, if a term, the associated content of a term, and an identification data set are in text form, in the process of determining the degree of match between a term and the field category to which the term belongs, a first text feature of the associated content of the term and a second text feature of data in the identification data set of the field category to which the term belongs can be extracted; by analyzing the degree of association between the first text feature and the second text feature, the degree of association can ultimately be converted into a degree of match between the term and the field category to which the term belongs.

[0086] Of course, the method for calculating the degree of match between a term and the category of the field to which the term belongs in the embodiment of the present application is not limited to the above method. For example, the degree of match between a term and the category of the field to which the term belongs can also be determined based on the frequency by simply determining the frequency of occurrence of a reference structured term in the associated content of the term in the recognition data set.

[0087] In summary, in the embodiments of the present application, the associated content of a term can greatly expand the dimension of the term, so that the definition scope of the term is not limited to one term, but the definition scope of the term is greatly expanded by the associated content with a wider breadth. Similarly, the identification data set can also greatly expand the dimension of the corresponding field category. In the process of determining the degree of match using the associated content of the term and the identification data set of the field category to which the term belongs, the associated content and the identification data set can provide richer semantic information, and a large amount of semantic information helps to improve the accuracy of calculating the degree of match, thereby improving the accuracy of quality management of term data. In addition, the entire quality management strategy can be implemented by automated information mining, information analysis, and information comparison, thereby greatly reducing manual participation, improving generation efficiency, and reducing production costs.

[0088] Reference Figure 8 , shows a flowchart of the specific steps of an embodiment of a text processing method of the present application.

[0089] Step 201: Obtain terms of at least one field category.

[0090] Specifically, this step may refer to the description in the above step 101, which will not be repeated here.

[0091] Step 202: According to the term, obtain the associated web page data as the associated content.

[0092] In the embodiment of the present application, it is possible to rely on abundant network resources to search for data associated with the term as associated content from the network resources, among which the more common ones are web resources provided by the website, and network resources include but are not limited to web resources, word library resources, database resources, etc.

[0093] Optionally, step 202 may include:

[0094] Sub-step 2021: According to the term, obtain the web page data from the target knowledge website as the associated content.

[0095] In an embodiment of the present application, the associated content may include web page data obtained from a target knowledge website. The target knowledge website may be an encyclopedia knowledge website, a proprietary domain knowledge website, etc. The data in the target knowledge website may be established in the format of entry-entry content, so the term may be used as a query entry, the corresponding entry content may be obtained in the target knowledge website, and the web page data corresponding to the entry content may be used as the associated content.

[0096] Optionally, sub-step 2021 may include:

[0097] Sub-step A1: In at least one target knowledge website, using the term as a query term, obtaining a term page corresponding to the term.

[0098] Reference Figure 1 ,The associated content acquisition module of the term data verification server can use the term as the search term, search the corresponding term page in the target knowledge website in the knowledge website, and use part or all of the content in the corresponding term page as the associated content of the term.

[0099] For example, for the term "coronary heart disease" in the medical field, you can search for the entry page corresponding to the entry "coronary heart disease" in encyclopedia knowledge sites. The entry page may include the above-mentioned aliases, English names, text paragraphs used to describe the entry and other related information.

[0100] Sub-step 2022: According to the term, obtain the web page data from the target search engine as the associated content.

[0101] In an embodiment of the present application, the associated content may also include web page data obtained from a target search engine. The target search engine stores a large amount of web page data and can obtain web page links of associated web pages through corresponding query indexes. In an embodiment of the present application, terms can be used as query indexes to obtain corresponding query results in the target search engine, and the web page data in the query results can be used as associated content.

[0102] Optionally, sub-step 2022 may include:

[0103] Sub-step A2: querying with the term in at least one target search engine to obtain query results including a plurality of web page data.

[0104] Reference Figure 1 The associated content acquisition module of the term data verification server can use the term as a query word to query the associated web pages in the search engine. The corresponding query results can include multiple web page links associated with the term, and according to the logic of the search engine, the multiple web page links can be sorted in descending order according to the relevance between the content and the term.

[0105] Sub-step A3: selecting at least one target web page data from the query results as the associated content.

[0106] Specifically, the embodiments of the present application can filter the query results and use part or all of the content in the web pages corresponding to the web links remaining after filtering as the associated content of the term. For example, when filtering multiple web links, the web page corresponding to the web link with the highest correlation can be selected as the target web page data, and the remaining web links can be filtered, or the web pages corresponding to several web links with the highest correlation can be selected as the target web page data, and the remaining web links can be filtered.

[0107] Furthermore, since there may be a lot of noise data (in this application, it may be text irrelevant to the content of the term) in the target web page data related to the term obtained through the search engine, further noise data filtering can be performed on the selected target web page data. Specifically, the embodiment of the present application can pre-define some text features of the noise data, so as to establish a corresponding noise data template, and use the noise data template to find data in the target web page data whose text features match the text features of the noise data, and filter out these data as noise data, so as to ensure the data quality of the related content.

[0108] Step 203, the associated content is parsed according to a preset structured parsing rule to obtain structured terms to be analyzed and / or text paragraphs to be analyzed in the associated content; the structured terms to be analyzed include: terms displayed in a structured form on the page.

[0109] Optionally, step 203 may be implemented by acquiring the to-be-analyzed structured terms and / or to-be-analyzed text paragraphs from the web page data according to a structured parsing template corresponding to the web page data.

[0110] In an embodiment of the present application, the associated content may include an entry page obtained from a target knowledge website, and a web page obtained from a target search engine, and both the entry page and the web page are structured page data. Specifically, the actual demand of the embodiment of the present application for term verification is to expand the dimension of the term through the associated content having data in the form of terms and text paragraphs. Therefore, according to the actual demand of the embodiment of the present application for term verification, a structured parsing rule for page data can be pre-defined to parse the structured terms to be analyzed and / or the text paragraphs to be analyzed in the associated content, and a structured parsing template can be established according to the preset structured parsing rules, and the associated content is parsed by the structured parsing template to obtain the parsing result. The structured parsing rule can be set based on the position of the content to be parsed in the page data structure, or based on the code structure of the parsed content in the page data structure, and then the structured parsing rule is used to parse the content to be extracted from the position or code structure of the page data.

[0111] Specifically, for the entry page obtained from the target knowledge website, the page structure is relatively fixed. For example, in the HyperText Markup Language (HTML) code of a knowledge website a, the structured term to be analyzed may have a corresponding structured tag:

[0112] XX

[0113] <dd title＝"xx">

[0114] For example, in the knowledge website a, in the entry page corresponding to the term "coronary atherosclerotic heart disease", the alias "coronary heart disease" corresponding to the term can be a structured term to be analyzed, which is as follows in the HTML code:

[0115] Other Names

[0116] <dd title＝"冠心病">

[0117] The embodiment of the present application can add the structured tags in the above format into the structured parsing template to parse the entry page of the term, so as to use the words in the entry page containing the structured tags in this format as the structured terms to be analyzed.

[0118] In addition, in the HTML code of the knowledge website a, the text paragraph to be analyzed may have a corresponding paragraph tag:

[0119] <div class＝"para"label-module＝"para"> XXX

[0120] For example, in the knowledge website a, in the entry page corresponding to the term "coronary atherosclerotic heart disease", a text paragraph corresponding to the term: "Risk factors for coronary heart disease include modifiable risk factors and unmodifiable risk factors. Understanding and intervening in risk factors contributes to the prevention and treatment of coronary heart disease" can be a text paragraph to be analyzed, which is as follows in HTML code:

[0121] <div class＝"para"label-module＝"para"> The risk factors for coronary heart disease include modifiable risk factors and unmodifiable risk factors. Understanding and intervening in risk factors can help prevent and treat coronary heart disease.

[0122] The embodiment of the present application can add the paragraph tags in the above format into the structured parsing template to parse the entry page of the term, so as to use the paragraphs in the entry page containing the paragraph tags in this format as the text paragraphs to be analyzed.

[0123] It should be noted that for different knowledge websites, the definition of structured parsing rules may be different. The embodiment of the present application can establish corresponding structured parsing rules by analyzing the page structure effects of the knowledge website as the source of the associated content in advance, thereby achieving the purpose of differentiated processing. Similarly, the definition of structured parsing rules for different web pages obtained from the target search is more likely to be different, and the above-mentioned means of differentiated processing can also be applied to web pages obtained from the target search.

[0124] In addition, the above example is based on the case where the associated content is text. When the associated content contains other forms of data, the corresponding text information can also be extracted from other forms of data through corresponding analysis means. For example, when the associated content contains a large number of web page images and videos, optical character recognition (OCR) (Optical Character Recognition) can be used to extract the corresponding text from the image and video data.

[0125] For example, refer to Fig. 9 , which shows a flow chart of a matching degree solving process of the present application, for Figure 1 The medical, logistics and automotive terms shown are Fig. 9 The verification module of the terminology data verification server can first perform structured analysis on the related content of medical terms, logistics terms, and automobile terms, so as to obtain the analysis results of the three fields including the structured terms to be analyzed and / or the text paragraphs to be analyzed, so as to subsequently carry out the process of solving the matching degree between the terms and the categories in the fields to which they belong based on the analysis results and the identification data sets of each field.

[0126] Step 204 : determining the degree of match between the term and the field category to which the term belongs based on the structured term to be analyzed and / or the text paragraph to be analyzed and the identification data set of the field category to which the term belongs.

[0127] In the embodiments of this application, refer to Fig. 9 For the analysis results and identification data sets of related content in the medical field, logistics field, and automobile field, in the process of solving the matching degree between terms and the categories in the corresponding fields based on the analysis results and identification data sets, the matching degree can be calculated based on text features / machine learning models / structured terms.

[0128] Optionally, the recognition data set includes: preset N-gram relationship features between reference words; step 204 may include:

[0129] Sub-step 2041: when the associated content includes at least one text paragraph to be analyzed, obtain N-gram relationship features between words to be analyzed according to the text paragraph to be analyzed.

[0130] In a process of calculating the degree of match between a term and its corresponding field category, it can be achieved by matching the text paragraph to be analyzed in the associated content with the N-gram feature of the reference word preset in the recognition data set, wherein the reference n-gram preset in the recognition data set can be obtained by analyzing the reference text paragraph in the recognition data set.

[0131] Specifically, the n-gram feature is a text feature, which is defined as follows: if a sentence S consists of m words (w1w2w3…wm), then the n-gram is defined as: {wiwi+1…wi+n-1|1≤i≤m-n+1}. This definition means that the n-gram feature is the association relationship established by n consecutive words in a sentence.

[0132] Wherein, n can be an integer greater than or equal to 1, and 1-gram is a language feature containing 1 word. For example, for a text paragraph: "Myocardial angina and myocardial infarction cause serious consequences, how atherosclerosis, myocardial angina and myocardial infarction are formed and developed", each word can establish a 1-gram.

[0133] 2-gram is a language feature obtained by establishing a corresponding relationship between two consecutive words. For example, for the above text paragraph, 2-gram may include: "atherosclerosis-sclerosis", "sclerosis-myocardium", "myocardial-angina", etc. Similarly, 3-gram, 4-gram and other n-gram features can also be established in sequence.

[0134] It should be noted that the preset N-gram relationship features between reference words included in the recognition data set can be obtained by analyzing the reference text paragraphs included in the recognition data set. The reference text paragraphs can be recognized text paragraphs with high confidence for accurately describing the field. The reference text paragraphs can also be called seed texts. For the "medical" field category, the reference text paragraphs can include text paragraphs that authoritatively describe and define "medical". For example: "Medical is a Chinese word with two meanings: treatment and treatment of diseases", "Chinese medicine has a history of thousands of years, and this word has only appeared in recent decades. In fact, this is a new word for international integration. Previously, treatment was mostly used. However, medical treatment also includes health care content", "Medical accidents refer to accidents in which medical institutions and their medical personnel violate medical and health management laws, relevant regulations, departmental regulations, and diagnosis, treatment and nursing standards and routines in medical activities, and negligence will cause personal injury to patients." etc.

[0135] In an embodiment of the present application, n-gram features (including 1-gram, 2-gram, 3-gram...) of all reference text paragraphs of the field category can be extracted, and based on the extracted n-gram features of the reference text paragraphs, a reference word N-gram relationship feature library containing all extracted n-gram features is established.

[0136] Sub-step 2042: Match the N-gram relationship features between the words to be analyzed with the N-gram relationship features between the reference words to determine the degree of match between the term and the field category to which the term belongs.

[0137] Specifically, before performing the operation of matching the N-gram relationship features between words to be analyzed with the N-gram relationship features between reference words, the embodiment of the present application can first count the number of occurrences of each n-gram feature in a preset reference N-gram relationship feature library between words, and establish a corresponding relationship between each n-gram feature and the number of occurrences.

[0138] For example, referring to the example provided in sub-step 2041 for the reference text paragraph: "myocardial angina and myocardial infarction, how atherosclerosis, myocardial angina and myocardial infarction are formed and developed", the preset reference word N-gram relationship feature library can count the number of occurrences of each 1-gram, 2-gram, 3-gram..., such as, 1-gram "atherosclerosis" appears once, 1-gram "myocardium" appears twice; 2-gram "myocardial-angina" appears twice, 2-gram "atherosclerosis" appears once, and so on.

[0139] After establishing the correspondence between each n-gram feature and the number of occurrences, the number of each n-gram feature extracted from the associated text of the term can be queried from the above correspondence based on the n-gram features (including 1-gram, 2-gram, 3-gram...) extracted from the associated text of the term, and the number of each n-gram queried is added up. The sum result can be used as the matching degree between the term and the field category to which the term belongs.

[0140] Furthermore, when the sum is greater than or equal to a preset threshold, it can be considered that the term belongs to the field category and the data quality of the term is high.

[0141] For example, referring to the above example, the n-gram features extracted from the associated text of the term include 1-gram "atherosclerosis", 2-gram "myocardial-angina" and 2-gram "atherosclerosis-sclerosis", and these three n-gram features are queried in the preset reference word N-gram relationship feature library, and the corresponding occurrence numbers obtained are 1, 2, and 1, respectively, that is, the matching degree between the term and the field category to which the term belongs can be 4.

[0142] It should be noted that the above content describes the matching degree calculation process using Chinese as an example. Similarly, the same idea can be used to calculate the matching degree for other forms of language or symbols. In addition, in order to ensure that the matching degree between the term and the field category to which the term belongs is solved with high accuracy, the data volume of the associated content of the term and the data volume of the identification data set of the field category need to be high, that is, the data volume of the associated content and the identification data set can be required to be greater than the preset data volume threshold. Among them, each data volume threshold can be set according to the actual situation, and the embodiment of the present application does not limit this.

[0143] Optionally, the identification data set includes: a machine learning model; step 204 may include:

[0144] Sub-step 2043: when the associated content includes at least one text paragraph to be analyzed, obtaining input features according to the text paragraph to be analyzed.

[0145] Sub-step 2044: input the input features into the machine learning model to determine the degree of match between the term and the field category to which the term belongs.

[0146] In another implementation process of calculating the degree of match between terms and their corresponding field categories, it can be achieved by associating the text paragraphs to be analyzed in the content with the machine learning model preset in the recognition data set, wherein the machine learning model preset in the recognition data set can be trained based on the reference text paragraphs and reference structured terms in the recognition data set.

[0147] The recognition data set may include one or more machine learning models, such as a machine learning model may include a language model, a domain vector space model, a classification model based on the K-means clustering algorithm (k-means) or the k-nearest neighbor algorithm (k-nn), etc.

[0148] The machine learning model can be understood as a mathematical model. A mathematical model is a scientific or engineering model constructed using mathematical logic methods and mathematical language. A mathematical model is a mathematical structure that is generally or approximately expressed using mathematical language with reference to the characteristics or quantitative dependencies of a certain system of things. This mathematical structure is a pure relational structure of a certain system depicted with the help of mathematical symbols. A mathematical model can be one or a group of algebraic equations, differential equations, difference equations, integral equations or statistical equations and their combinations, which quantitatively or qualitatively describe the relationship or causal relationship between the variables of the system. In addition to mathematical models described by equations, there are also models described by other mathematical tools, such as algebra, geometry, topology, mathematical logic, etc. Among them, the mathematical model describes the behavior and characteristics of the system rather than the actual structure of the system. Among them, the simulator adopts machine learning, deep learning methods, etc. for model training. Machine learning methods may include linear regression, decision tree, random forest, extreme gradient boosting (xgboost, eXtreme Gradient Boosting), light gradient booster (lightgbm, Light Gradient Boosting Machine), k-means classification or k-nn classification, etc. Deep learning methods may include convolutional neural networks (Convolutional Neural Networks, CNN), long short-term memory networks (Long Short-Term Memory, LSTM), gated recurrent units (Gated Recurrent Unit, GRU), etc.

[0149] Specifically, the recognition data set includes a language model (such as an n-gram model), a domain vector space model, and a classification model based on a K-means clustering algorithm (k-means) or a k-nearest neighbor algorithm (k-nn).

[0150] In one implementation, the machine learning model may be a domain vector space model, which is a classification model. Since different texts can be expressed through their respective vector features, the vector distance between vector features can reflect the degree of semantic association between texts. Therefore, the embodiment of the present application may use the domain vector space model to determine the degree of match between a term and the domain category to which the term belongs.

[0151] The domain vector space model can be trained based on reference text paragraphs and reference structured terms in the identification data set of the domain category. The training goal is to use the vector features of the associated content of the input term as input features, output the probability that the term belongs to the domain category through the calculation of the vector distance, and use the probability as the matching degree between the term and the domain category to which the term belongs. During the training process, the domain vector space model can establish a validation set based on the vector features of the reference text paragraphs and the reference structured terms. In practical applications, the matching degree between the term and the domain category to which the term belongs can be obtained by calculating the vector distance between the input feature and the vector features in the validation set.

[0152] For example, the embodiment of the present application can extract the vector features of the associated content and each text paragraph to be analyzed; calculate the vector distance between the associated content and each text paragraph to be analyzed based on the vector features of the associated content and the text paragraph to be analyzed; and select a target vector distance greater than or equal to a preset distance threshold from all vector distances; determine a matching value that reflects the degree of matching between the term text and the field category corresponding to the term text based on the target vector distance, and determine the field category to which the term text belongs when the matching value is greater than or equal to a preset first matching threshold. Among them, each text paragraph to be analyzed can be set with a weight according to the accuracy of the data. When there are multiple target vector distances, each target vector distance can be weighted averaged according to the weight of the text paragraph to be analyzed corresponding to each target vector distance, and the weighted average finally obtained can be the matching value.

[0153] In addition, the machine learning model can be a neural network model, and the neural network model can also realize the function of a classification model, that is, the training goal of the neural network model is to use the convolution feature based on the associated content of the input term as the input feature, and through the calculation of the convolution feature, output the probability that the term belongs to the field category, and use the probability as the match between the term and the field category to which the term belongs.

[0154] In one implementation, the machine learning model may be a classification model based on a k-means or k-nn method, or a classification model based on a binary classification method. The classification model can output the category of the term considered by the model, or the probability of the term belonging to each category, based on the classification method it adopts and the characteristics of the associated content of the term, so as to obtain the matching degree between the term and the category of the field to which the term belongs according to these output results.

[0155] Optionally, the identification data set includes: preset reference structured terms; step 204 may include:

[0156] Sub-step 2045: When the associated content includes at least one structured term to be analyzed, match the structured term to be analyzed with the reference structured term to determine the degree of match between the term and the field category to which the term belongs.

[0157] In an embodiment of the present application, the preset reference structured terms included in the recognition data set may be reference structured terms predefined for the field category, and the reference structured terms may be recognized proprietary terms or high-quality terms within the field category. For example, for the "medical" field category, the reference structured terms may include: "treatment", "disease", "health", "treatment" and other terms.

[0158] In one implementation, an embodiment of the present application may calculate the number of reference structured terms included in all structured terms to be analyzed in the associated content, and the degree of match between the terms and the field category to which the terms belong, and when the matching value is greater than or equal to a second preset quantity threshold, determine that the term belongs to the field category.

[0159] For example, the structured terms to be analyzed in the associated content of the term "coronary heart disease" include "visiting a doctor", "medical treatment", and "treatment". For the above-mentioned preset reference structured terms "treatment", "disease", "health", and "treatment", a matching value of 2 can be obtained.

[0160] Optionally, the method may further include:

[0161] Step 205: Obtain sample text belonging to the domain category.

[0162] In the embodiments of this application, refer to Figure 1 The sample texts used as the data source of the identification data set of the domain category can also be obtained from websites, thesaurus, and databases. Since the identification data set exists as a function similar to the verification set, the requirements for its data quality are relatively high. In one case, the high data quality of the data in the identification data set can be guaranteed, thereby meeting the precision requirements of the embodiment of the present application for the matching degree between the term and the domain category to which the term belongs. In another case, a large amount of data in the identification data set can be guaranteed, thereby meeting the precision requirements of the embodiment of the present application for the matching degree between the term and the domain category to which the term belongs.

[0163] Specifically, the process of obtaining the sample text can also refer to the above embodiment. Figure 1 The process of obtaining the associated text may be performed from data sources such as a vocabulary, a database, a knowledge website, a search engine, etc. In addition, the sample text may also be obtained by developers through the definition of domain categories, which is not limited in the embodiments of the present application.

[0164] Step 206: construct a recognition data set of the field category based on the sample text.

[0165] In one implementation, the process of constructing the recognition data set of the field category from the sample text can refer to the content of the above-mentioned step 203, that is, by parsing the sample text according to the preset structured parsing rules, obtaining the reference structured terms and / or the reference text paragraphs to be analyzed in the sample text, and directly constructing the reference structured terms and the reference text paragraphs to be analyzed into the recognition data set, or constructing the text features (vector features, n-gram features) of the reference structured terms and the reference text paragraphs to be analyzed into the recognition data set.

[0166] In another implementation, a training set can be constructed from sample texts, and a machine learning model such as in sub-step 2044 can be constructed as a recognition data set to achieve the purpose of determining the degree of match between a term and the field category to which the term belongs through the machine learning model.

[0167] In summary, in the embodiments of the present application, the associated content of a term can greatly expand the dimension of the term, so that the definition scope of the term is not limited to one term, but the definition scope of the term is greatly expanded by the associated content with a wider breadth. Similarly, the identification data set can also greatly expand the dimension of the corresponding field category. In the process of determining the degree of match using the associated content of the term and the identification data set of the field category to which the term belongs, the associated content and the identification data set can provide richer semantic information, and a large amount of semantic information helps to improve the accuracy of calculating the degree of match, thereby improving the accuracy of quality management of term data. In addition, the entire quality management strategy can be implemented by automated information mining, information analysis, and information comparison, thereby greatly reducing manual participation, improving generation efficiency, and reducing production costs.

[0168] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0169] On the basis of the above embodiments, this embodiment further provides a text processing device, which is applied to electronic devices such as terminal devices and server terminals.

[0170] Reference Fig.10 , shows a structural block diagram of a text processing device embodiment of the present application, which may specifically include the following modules:

[0171] A first acquisition module 301 is used to acquire terms of at least one field category;

[0172] The second acquisition module 302 is used to acquire the associated content of the term;

[0173] Optionally, the second acquisition module 302 includes:

[0174] The acquisition submodule is used to acquire the associated web page data as the associated content according to the term.

[0175] Optionally, the acquisition submodule includes:

[0176] A third acquisition unit is used to acquire the web page data as the associated content from the target knowledge website according to the term;

[0177] Optionally, the third acquiring unit includes:

[0178] The first acquisition subunit is used to acquire the entry page corresponding to the term in at least one target knowledge website by taking the term as a query entry.

[0179] And / or, a fourth acquisition unit is used to acquire the web page data as the associated content from a target search engine according to the term.

[0180] Optionally, the fourth acquiring unit includes:

[0181] A second acquisition subunit is used to perform a query using the term in at least one target search engine to obtain a query result including a plurality of web page data;

[0182] The determination subunit is used to select at least one target web page data from the query result as the associated content.

[0183] The verification module 303 is used to determine the matching degree between the term and the field category to which the term belongs based on the associated content of the term and the identification data set of the field category to which the term belongs.

[0184] Optionally, the verification module includes:

[0185] The parsing submodule is used to parse the associated content according to a preset structured parsing rule to obtain structured terms to be analyzed and / or text paragraphs to be analyzed in the associated content; the structured terms to be analyzed include: terms displayed in a structured form on the page;

[0186] Optionally, when the associated content is web page data, the parsing submodule includes:

[0187] The parsing unit is used to obtain the structured terms to be analyzed and / or the text paragraphs to be analyzed from the web page data according to the structured parsing template corresponding to the web page data.

[0188] The verification submodule is used to determine the matching degree between the term and the field category to which the term belongs based on the structured term to be analyzed and / or the text paragraph to be analyzed and the identification data set of the field category to which the term belongs.

[0189] Optionally, the recognition data set includes: preset N-gram relationship features between reference words; the verification submodule includes:

[0190] A first acquisition unit is used to acquire N-gram relationship features between words to be analyzed according to the text paragraph to be analyzed when the associated content includes at least one text paragraph to be analyzed;

[0191] The first verification unit is used to match the N-gram relationship feature between the words to be analyzed with the N-gram relationship feature between the reference words to determine the matching degree between the term and the field category to which the term belongs.

[0192] Optionally, the identification data set includes: a machine learning model; the verification submodule includes:

[0193] A second acquisition unit, configured to acquire input features according to the text paragraph to be analyzed when the associated content includes at least one text paragraph to be analyzed;

[0194] The second verification unit is used to input the input features into the machine learning model to determine the matching degree between the term and the field category to which the term belongs.

[0195] Optionally, the identification data set includes: preset reference structured terms; the verification submodule includes:

[0196] The third verification unit is used to match the structured term to be analyzed with the reference structured term when the associated content includes at least one structured term to be analyzed, and determine the matching degree between the term and the field category to which the term belongs.

[0197] Optionally, the device further comprises:

[0198] A third acquisition module is used to acquire sample text belonging to the field category;

[0199] A construction module is used to construct a recognition data set of the field category according to the sample text.

[0200] In summary, in the embodiments of the present application, the associated content of a term can greatly expand the dimension of the term, so that the definition scope of the term is not limited to one term, but the definition scope of the term is greatly expanded by the associated content with a wider breadth. Similarly, the identification data set can also greatly expand the dimension of the corresponding field category. In the process of determining the degree of match using the associated content of the term and the identification data set of the field category to which the term belongs, the associated content and the identification data set can provide richer semantic information, and a large amount of semantic information helps to improve the accuracy of calculating the degree of match, thereby improving the accuracy of quality management of term data. In addition, the entire quality management strategy can be implemented through automated information mining, information analysis, and information comparison, thereby greatly reducing manual participation, improving generation efficiency, and reducing production costs.

[0201] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0202] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and server terminals (clusters).

[0203] The embodiments of the present disclosure may be implemented as a device that uses any appropriate hardware, firmware, software, or any combination thereof to perform desired configurations, and the device may include electronic devices such as terminal devices and server (cluster). Fig.11 An exemplary device 700 that can be used to implement various embodiments described in the embodiments of the present application is schematically shown.

[0204] For one embodiment, Fig.11 An exemplary apparatus 700 is shown having one or more processors 702, a control module (chip set) 704 coupled to at least one of the processor(s) 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 708 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0205] The processor 702 may include one or more single-core or multi-core processors, and the processor 702 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 700 can be used as a terminal device, a server (cluster), and other devices described in the embodiments of the present application.

[0206] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage device 708) having instructions 714 and one or more processors 702 combined with the one or more computer-readable media and configured to execute the instructions 714 to implement a module to perform the actions described in the present disclosure.

[0207] For one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 702 and / or any suitable device or component in communication with the control module 704 .

[0208] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0209] The memory 706 may be used, for example, to load and store data and / or instructions 714 for the device 700. For one embodiment, the memory 706 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 706 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0210] For one embodiment, control module 704 may include one or more input / output controllers to provide an interface to NVM / storage device 708 and input / output device(s) 710 .

[0211] For example, NVM / storage 708 may be used to store data and / or instructions 714. NVM / storage 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0212] NVM / storage device 708 may include storage resources that are physically part of the device on which apparatus 700 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 708 may be accessed via input / output device(s) 710 over a network.

[0213] (One or more) input / output devices 710 may provide an interface for the apparatus 700 to communicate with any other appropriate device, and the input / output device 710 may include a communication component, an audio component, a sensor component, etc. The network interface 712 may provide an interface for the apparatus 700 to communicate through one or more networks, and the apparatus 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0214] For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 704. For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers of the control module 704 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704. For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704 to form a system-on-chip (SoC).

[0215] In various embodiments, the device 700 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 700 may have more or fewer components and / or different architectures. For example, in some embodiments, the device 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0216] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0217] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0218] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0219] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0220] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0221] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0222] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0223] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0224] The above is a detailed introduction to a text processing method, device, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A text processing method, characterized in that: The method comprises: Get at least one term of the domain category; Obtaining content associated with the term; The degree of match between the term and the field category to which the term belongs is determined based on the associated content of the term and the identification data set of the field category to which the term belongs; wherein the associated content of the term includes: at least one of the synonyms, antonyms, aliases, Chinese / English names of the term and text paragraphs explaining the term; the identification data set includes reference structured terms or reference text paragraphs pre-defined for the field category to which the term belongs.

2. The method according to claim 1, characterized in that The determining the degree of match between the term and the field category to which the term belongs based on the associated content of the term and the identification data set of the field category to which the term belongs includes: Parsing the associated content according to preset structured parsing rules to obtain structured terms to be analyzed and / or text paragraphs to be analyzed in the associated content; the structured terms to be analyzed include: terms displayed in a structured form on the page; The degree of matching between the term and the field category to which the term belongs is determined based on the structured term to be analyzed and / or the text paragraph to be analyzed, and the identification data set of the field category to which the term belongs.

3. The method according to claim 2, characterized in that In the case where the associated content is web page data, parsing the associated content according to a preset structured parsing rule to obtain structured terms to be analyzed and / or text paragraphs to be analyzed in the associated content includes: The structured terms to be analyzed and / or the text paragraphs to be analyzed are obtained from the web page data according to the structured parsing template corresponding to the web page data.

4. The method according to claim 2, characterized in that: The recognition data set includes: preset N-gram relationship features between reference words; Determining the degree of match between the term and the field category to which the term belongs based on the structured term to be analyzed and / or the text paragraph to be analyzed and the identification data set of the field category to which the term belongs includes: In a case where the associated content includes at least one text paragraph to be analyzed, obtaining N-gram relationship features between words to be analyzed according to the text paragraph to be analyzed; The N-gram relationship feature between the words to be analyzed is matched with the N-gram relationship feature between the reference words to determine the matching degree between the term and the field category to which the term belongs.

5. The method according to claim 2, characterized in that: The identification data set includes: a machine learning model; Determining the degree of match between the term and the field category to which the term belongs based on the structured term to be analyzed and / or the text paragraph to be analyzed and the identification data set of the field category to which the term belongs includes: In a case where the associated content includes at least one text paragraph to be analyzed, acquiring input features according to the text paragraph to be analyzed; The input features are input into the machine learning model to determine the matching degree between the term and the field category to which the term belongs.

6. The method according to claim 2, characterized in that The identification data set includes: preset reference structured terms; Determining the degree of match between the term and the field category to which the term belongs based on the structured term to be analyzed and / or the text paragraph to be analyzed and the identification data set of the field category to which the term belongs includes: In the case that the associated content includes at least one structured term to be analyzed, the structured term to be analyzed is matched with the reference structured term to determine the degree of match between the term and the field category to which the term belongs.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: Obtaining sample text belonging to the domain category; Based on the sample text, a recognition data set of the field category is constructed.

8. The method according to any one of claims 1 to 6, characterized in that: The obtaining of the associated content of the term includes: According to the term, associated web page data is acquired as associated content.

9. The method according to claim 8, characterized in that The step of obtaining the associated webpage data as the associated content according to the term includes: According to the term, acquiring the web page data from a target knowledge website as the associated content; And / or, according to the term, obtaining the web page data from a target search engine as the associated content.

10. The method according to claim 9, characterized in that According to the term, acquiring the web page data from the target knowledge website as the associated content includes: In at least one target knowledge website, the term is used as a query term to obtain a term page corresponding to the term.

11. The method according to claim 9, characterized in that The acquiring the webpage data as the associated content from a target search engine according to the term includes: In at least one target search engine, querying with the term to obtain query results including a plurality of web page data; At least one target web page data is selected from the query results as the associated content.

12. A text processing device, characterized in that: The device comprises: A first acquisition module, used to acquire terms of at least one field category; A second acquisition module is used to acquire the associated content of the term; A verification module is used to determine the degree of match between the term and the field category to which the term belongs based on the associated content of the term and the identification data set of the field category to which the term belongs; wherein the associated content of the term includes: at least one of the synonyms, antonyms, aliases, Chinese / English names of the term and text paragraphs explaining the term; the identification data set includes reference structured terms or reference text paragraphs pre-defined for the field category to which the term belongs.

13. The device according to claim 12, characterized in that The verification module comprises: The parsing submodule is used to parse the associated content according to a preset structured parsing rule to obtain structured terms to be analyzed and / or text paragraphs to be analyzed in the associated content; the structured terms to be analyzed include: terms displayed in a structured form on the page; The verification submodule is used to determine the matching degree between the term and the field category to which the term belongs based on the structured term to be analyzed and / or the text paragraph to be analyzed and the identification data set of the field category to which the term belongs.

14. The device according to claim 13, characterized in that In the case where the associated content is web page data, the parsing submodule includes: The parsing unit is used to obtain the structured terms to be analyzed and / or the text paragraphs to be analyzed from the web page data according to the structured parsing template corresponding to the web page data.

15. The device according to claim 13, characterized in that The recognition data set includes: preset N-gram relationship features between reference words; the verification submodule includes: A first acquisition unit is used to acquire N-gram relationship features between words to be analyzed according to the text paragraph to be analyzed when the associated content includes at least one text paragraph to be analyzed; The first verification unit is used to match the N-gram relationship feature between the words to be analyzed with the N-gram relationship feature between the reference words to determine the matching degree between the term and the field category to which the term belongs.

16. The device according to claim 13, characterized in that The identification data set includes: a machine learning model; the verification submodule includes: A second acquisition unit, configured to acquire input features according to the text paragraph to be analyzed when the associated content includes at least one text paragraph to be analyzed; The second verification unit is used to input the input features into the machine learning model to determine the matching degree between the term and the field category to which the term belongs.

17. The device according to claim 13, characterized in that The identification data set includes: preset reference structured terms; the verification submodule includes: The third verification unit is used to match the structured term to be analyzed with the reference structured term when the associated content includes at least one structured term to be analyzed, and determine the matching degree between the term and the field category to which the term belongs.

18. The device according to any one of claims 12 to 17, characterized in that The device also includes: A third acquisition module is used to acquire sample text belonging to the field category; A construction module is used to construct a recognition data set of the field category according to the sample text.

19. The device according to any one of claims 12 to 17, characterized in that The second acquisition module includes: The acquisition submodule is used to acquire the associated web page data as the associated content according to the term.

20. The device according to claim 19, characterized in that The acquisition submodule includes: A third acquisition unit is used to acquire the web page data as the associated content from the target knowledge website according to the term; And / or, a fourth acquisition unit is used to acquire the web page data as the associated content from a target search engine according to the term.

21. The device according to claim 20, characterized in that The third acquisition unit includes: The first acquisition subunit is used to acquire the entry page corresponding to the term in at least one target knowledge website by taking the term as a query entry.

22. The device according to claim 20, characterized in that The fourth acquiring unit includes: A second acquisition subunit is used to perform a query using the term in at least one target search engine to obtain a query result including a plurality of web page data; The determination subunit is used to select at least one target web page data from the query result as the associated content.

23. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, cause the processor to perform the method according to one or more of claims 1-11.

24. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the method of one or more of claims 1-11.

Images