Text root determination method and device, computer equipment and storage medium

Abstract

The invention discloses a method and a device for determining a root of a text, computer equipment and a storage medium. In the application, candidate roots corresponding to the text are searched from a root library; then according to the word frequency inverse document frequency of the candidate word root, the part-of-speech information, the document information of the document where the candidate word root is located, and the position information of the candidate word root in the document, determining the weight parameter of the candidate word root, and finally, according to the weight parameter of the candidate word root, determining the position of the candidate word root in the document. And determining a target word root corresponding to the text from the candidate word roots. According to the method and the device, the accuracy and uniformity of text root determination can be improved, and the accuracy and normalization of data table contents are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, computer equipment and storage medium for determining a word root of a text. Background Art

[0002] With the continuous development of big data management technology, data from different data sources are integrated and stored in data warehouses in the form of data tables to facilitate the management and query of massive data. Data warehouse model table design is an important part of the data warehouse development stage.

[0003] When designing a data warehouse model table, a large number of existing data tables will be referenced. Due to the inconsistent field design specifications of different data tables, it is difficult to determine accurate and unified root words for the text of business requirements, which makes the data in the data warehouse model table prone to errors and confusion. Summary of the invention

[0004] In response to the above technical problems, embodiments of the present application provide a method, apparatus, computer device and storage medium for determining word roots of a text.

[0005] In a first aspect, an embodiment of the present application provides a method for determining a root of a text, comprising:

[0006] Search for candidate word roots corresponding to the text from the word root library;

[0007] Determine a weight parameter of the candidate root according to the inverse document frequency of the candidate root, part of speech information, document information of the document where the candidate root is located, and position information of the candidate root in the document;

[0008] According to the weight parameters of the candidate word roots, a target word root corresponding to the text is determined from the candidate word roots.

[0009] In a second aspect, an embodiment of the present application provides a text root determination device, comprising:

[0010] A candidate root word search module is used to search for candidate root words corresponding to a text from a root word library;

[0011] A weight parameter determination module, used to determine the weight parameter of the candidate root according to the word frequency inverse document frequency, part of speech information, document information of the document where the candidate root is located, and position information of the candidate root in the document;

[0012] The target root word determination module is used to determine the target root word corresponding to the text from the candidate root words according to the weight parameters of the candidate root words.

[0013] In a third aspect, an embodiment of the present application further provides a computer device, including a memory storing a plurality of instructions; a processor loads the instructions from the memory to execute the steps of any text root determination method provided in the embodiment of the present application.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the steps of any text root determination method provided in an embodiment of the present application.

[0015] In a fifth aspect, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the root determination method of any text provided in the embodiment of the present application.

[0016] Adopting the scheme of the embodiment of the present application, the candidate root corresponding to the text is searched from the root library, and the weight parameter of the candidate root is determined according to the word frequency inverse document frequency, part of speech information, document information of the document where the candidate root is located, and the position information of the candidate root in the document, so that the self information of each candidate root can be considered in combination with the document information where it is located, and the rational weight of each candidate root in the root determination of the text can be determined more comprehensively and accurately on the basis of the word frequency inverse document frequency, thereby improving the accuracy of the root determination of the text. Finally, according to the weight parameter of the candidate root, the target root corresponding to the text is determined from the candidate root. In this way, a most accurate and non-repetitive target root can be determined for the roots of different texts, ensuring the uniformity of the matching roots, thereby also improving the accuracy and standardization of the generated data table content. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a schematic diagram of the application environment of the text root determination method provided in the embodiment of the present application;

[0019] Figure 2 It is a schematic diagram of an embodiment of a method for determining a root word of a text provided in an embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of an embodiment of a method for determining a root word of a text provided in an embodiment of the present application;

[0021] Figure 4 It is a schematic diagram of the overall flow of existing model table design;

[0022] Figure 5 It is a schematic diagram of the overall flow of model table design provided in the embodiments of the present application;

[0023] Figure 6 is an exemplary ER diagram drawn in an embodiment of the present application;

[0024] Figure 7 is a schematic diagram of obtaining text through user operation provided in an embodiment of the present application;

[0025] Figure 8 It is a schematic diagram of determining a target root word through a root word of a text provided in an embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of determining the position of a candidate root in a document provided in an embodiment of the present application;

[0027] Fig.10 It is a schematic diagram of the structure of a text root determination device provided in an embodiment of the present application;

[0028] Fig.11 It is a schematic diagram of the internal structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0030] The root determination method of the text in one embodiment of the present application can be run on a local terminal device or a server. When the game interaction method is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0031] In order to better understand the root determination method, device, electronic device and storage medium of a text provided in the embodiment of the present application, the application environment applicable to the embodiment of the present application is described below.

[0032] See also Figure 1 , Figure 1 A schematic diagram of an application environment of the text root determination method provided by an embodiment of the present application is shown. As an implementation mode, the text root determination method provided by the embodiment of the present application can be applied to an electronic device. The electronic device can be, for example, Figure 1 The server 110 shown in the figure can communicate with the terminal device 120 through a network. The network is a medium for providing a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as wired communication links, wireless communication links, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the electronic device can also be a smart phone, a laptop computer, etc.

[0033] It should be understood that Figure 1 The server 110, network and terminal device 120 are merely illustrative, and any number of servers, networks and terminal devices may be provided as required. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the terminal device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 at the same time.

[0034] In some embodiments, the terminal device 120 may obtain text data input by the user. Further, the terminal device 120 sends the text data input by the user to the server 110 through the network. After the server 110 receives the text data of the user, the text data may be processed by the text root determination method described in the embodiment of the present application.

[0035] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is a terminal device as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0036] The text root determination method of this embodiment can be applied to any scenario that needs to be executed, and the method can be applied to products in these scenarios, such as data table design systems, text processing systems, news classification systems, etc. Before introducing the text root determination method of this embodiment, the relevant technical data in the common scenarios of the text root determination method of this embodiment are first explained.

[0037] First of all, data warehouse is abbreviated as data warehouse. Data warehouse is a centralized repository for integrating, storing and managing large amounts of structured and unstructured data. The data table of the data warehouse is a model table, which is a structured way to describe and organize data in the data warehouse. Common model tables include dimension tables, fact tables, and zipper tables (Slowly Changing Dimension, SCD). Among them, the dimension table is used to describe the contextual information of the indicators (facts) in the fact table, such as sales, quantity, etc. The fact table stores a large amount of business fact data to support various types of query and analysis operations. The zipper table is a special dimension table used to track changes in dimension data over time.

[0038] Take this method applied to the data table design system of the data warehouse as an example, please refer to Figure 4 , Figure 4 It is a schematic diagram of the overall process of existing model table design. Figure 4 The scenarios include data sources, data warehouse ODS layer, data warehouse CDM layer model table design system and model table users. Among them, the data warehouse ODS layer (Operational Data Store) is usually located at the core of the data warehouse architecture, responsible for receiving real-time or near real-time data from various business systems and data sources, but the data warehouse ODS layer will not make any modifications to the data, such as data cleaning and noise reduction, and always keep the data consistent with the data source.

[0039] Among them, the data warehouse CDM layer (Conformed Dimensional Model) is used to store shared dimensions and attributes related to the business, as well as the relationship between these dimensions and attributes, to provide a consistent dimensional view for the data warehouse. The data warehouse CDM layer consists of two parts: the data warehouse detailed layer (DWD layer) and the data warehouse summary layer (DWS layer). Among them, the DWD layer refers to the data detail layer, which is used to receive the original data from the data warehouse ODS layer, and perform operations such as cleaning, standardization, dimension degradation, and abnormal data removal for unified processing to provide support for data analysis. The main function of the DWS layer is to aggregate and summarize the data in the DWD layer according to the theme to form a wide table, thereby improving data analysis performance. According to the functions and characteristics of the data warehouse CDM layer, integrating the model table design system in the data warehouse CDM layer for model table design can improve the consistency, comparability and quality of data table design.

[0040] In the overall process of existing model table design, the data from different data sources is first integrated through the data warehouse ODS layer. The data table design system of the data warehouse CDM layer can analyze the access data in combination with the business scenario, confirm the data table that needs to be referenced, and draw an ER diagram to help model table designers clarify the relationship between business logic and data. Figure 6 The following is an example of an ER diagram that has been drawn. In the online shopping business scenario, users and orders are two entities that have a corresponding relationship, and each of them has multiple attributes. For example, the attributes of users include user name, phone number, user information, etc., and the attributes of orders include order number, amount, etc. Then, in the future, the model table can be designed based on the relationship between the above entities and attributes. For example, the primary key of the user table can be designed as the user number (User ID), and the lower layer includes fields such as user name and password. After the model table design is completed, the code development process of writing the program can be carried out, and the data link can be established and sent to the downstream model table users.

[0041] It can be seen that in the process of model table design, since the data warehouse CDM layer needs to reference multiple data tables from the access data, which may contain hundreds or even thousands of fields, designers need to confirm and understand the meaning of each field one by one. There will be fields with the same or similar meanings, and it is necessary to confirm with the data source whether the meaning of the field is correct and observe the actual data corresponding to the field to determine how to use the field, making the overall process of model table design time-consuming and inefficient.

[0042] In addition, since model table design is usually not completed by a single designer, it may be difficult for multiple designers to form a consistent concept of the root of certain texts. For example, for the text "user", there may be disagreements on whether to use the root "user_name" or "username". For some common field roots, a standardized comparison table can be formed through conventions, and the mapping relationship between text and root can be defined according to this comparison table. However, when encountering some uncommon field roots, it is difficult to form a unified specification, resulting in the same root having multiple different names in the data warehouse CDM layer, which will cause confusion in the root definition and the inability to correctly match the text and root.

[0043] It should also be noted that the inconsistent matching standards between the text and word roots of the model table will also lead to confusion in the use of the model table, which will directly affect the smooth development of the business. For example, credit reporting, regulatory reporting and other businesses have extremely high requirements for the accuracy and timeliness of the data table content, and no errors are allowed. If there is confusion and inconsistent field naming in the model table, it will directly affect the normal operation of these businesses.

[0044] See also Figure 5 In the embodiment of the present application, a root library is added to the model table design system of the data warehouse CDM layer. The root library maintains the Chinese text and English root information of the fields that access the data. For example, for the user name field, the Chinese text is "user name" and the English root can be "user_name". During the ER diagram (Entity-Relationship Diagram) drawing stage, the model table design system can perform root determination operations on the texts required in the design process in the root library, determine a unique and accurate root for each text, and add the successfully matched text and root to the model table. If the match is unsuccessful, continue to improve the root library and then re-perform root determination operations on the text. Figure 5 As shown, drawing the ER diagram and performing text operations in the root library is a process of mutual confirmation. Therefore, a double arrow is used to connect the two in the figure. When drawing the ER diagram, it is necessary to determine the correct root for the text in the root library. Correspondingly, the matching root in the root library can be filled in the ER diagram, thereby improving the accuracy of the ER diagram and improving the model table design.

[0045] In the above way, based on the existing text information and root information in the root library, the time of model table designers to understand the meaning of different fields can be saved, which greatly improves the efficiency of model table design. Then, through the root determination operation of the root library, a unique and accurate root can be given to each text, so that the text and the root have a one-to-one correspondence, forming a unified root determination specification, reducing the confusion of text and root definitions and matching errors, thereby improving the accuracy, standardization and efficiency of model table design, and avoiding the adverse business impact on the model table users due to incorrect model table content.

[0046] Please refer to Figure 2 The specific process of the text root determination method provided in the embodiment of the present application may include the following steps 201 to 203, which are specifically introduced below.

[0047] Step 201, searching for candidate word roots corresponding to the text from a word root library.

[0048] Among them, text refers to the text content to be subjected to the root determination operation, and the root library refers to the database or system that stores and maintains the roots. In the scenario of this application, text can refer to the field text used in the model table design phase. For example, it can specifically refer to the name or description of the first language type (such as Chinese) of the field. The name or description of the field needs to be converted into the root of the second language type (such as English) later to meet the design language of programming development. For example, if a user data table is designed, and its business requirement is to display the relevant information of the user, the text involved may include "user name", "name", "age", etc., and the corresponding roots may include "user", "name", "age", etc.

[0049] It can be understood that the content of the model table is designed based on business requirements, such as Figure 6 The ER diagram in the figure shows the business logic of online shopping. Based on the ER diagram, you can determine the text used in designing the model table. For example, you may use texts such as "user" and "order".

[0050] In the scenario of the present application, since some common and common texts can directly determine the corresponding root words, the root word determination operation can be performed only for texts with uncertain meanings or with specific requirements. Therefore, in some embodiments, the text can be obtained by receiving user input content.

[0051] See also Figure 7 The designer of the model table can use the user end to log in to the model table design system and input text in the form of text query. During the process of the designer inputting text, the text query interface can display the existing text in the root word library. Figure 7As shown, when the designer inputs "yong", the query interface can display texts containing the character "yong", such as "user", "user name", "yonghu name", etc. in the form of a drop-down box, and monitor the designer's confirmation operation on the above texts. For example, if it is detected that the designer clicks to confirm the operation of "user name", it can be received. In this way, it can help the model table designer quickly and accurately lock the text and reduce the situation of text input errors.

[0052] Among them, the candidate root is an item in the root library that can perform a confirmation operation with the text, and there is the same or similar meaning between the candidate root and the text. For example, if the text is "user name", the candidate roots can be "user_name" and "username". Both of these candidate roots can be correctly matched with "user name". However, in order to ensure the unity of the text and root matching and the standardization of the model table design, the root determination operation in the embodiments of this application only determines a unique root for each text. Therefore, finally, one most suitable root for the text still needs to be determined from multiple candidate roots.

[0053] Step 202: Determine the weight parameter of the candidate root according to the term frequency-inverse document frequency, part-of-speech information, document information of the document where the candidate root is located, and position information of the candidate root in the document.

[0054] Among them, the term frequency-inverse document frequency is TF-IDF, which is short for "Term Frequency-Inverse Document Frequency". It is an algorithm for evaluating the importance of vocabulary and is divided into two parts: term frequency (TF) and inverse document frequency (IDF). For the root t, the specific algorithm of TF-IDF is as follows:

[0055] TF = the number of occurrences of the root t in the root library / the total number of all roots in the root library;

[0056] IDF = log (the total number of documents in the root library / the number of documents containing the root t + 1);

[0057] TF-IDF = TF × IDF.

[0058] It can be understood that the higher the TF-IDF value, the higher the importance of the root in the document and the root library, and at the same time, it can distinguish its particularity in the entire root library, so as to determine the importance of each candidate root in the root library and provide a preliminary direction for the root determination of the text.

[0059] The part-of-speech information may be relevant information expressing the part-of-speech of the root, such as noun, verb, adjective, etc. The part-of-speech of the candidate root may include noun, verb, adjective, etc. Candidate roots of different parts of speech carry different semantic information, for example, verbs represent actions or states, nouns represent entities or concepts, and adjectives describe properties. Therefore, in the specific design requirements of the model table, roots of certain parts of speech may be more critical and important. For example, in the data table related to the information extraction task, verbs play a key role in describing the behavior of the occurrence of events, and the part-of-speech weight of the candidate root of the verb part of speech is higher than that of the candidate root of the adjective part of speech and the noun part of speech.

[0060] It can be understood that since the roots in the root word library are obtained from different data sources, which may come from some data tables or documents, in order to ensure the accuracy of the root word library's understanding of the root word meaning and fully consider the important new factors affecting the root word by the document, the root word library can store not only the root word but also the documents of the root word, such as storing the root word a, and the documents A and B where the root word a is located. Therefore, the weight parameter of the candidate root word can be determined by combining the document information of the document where the candidate root word is located and the position information of the candidate root word in the document.

[0061] Among them, the weight parameter can be another parameter that reflects the importance of the root word other than the word frequency inverse document frequency. It can be understood that for each candidate root, only calculating the word frequency inverse document frequency may not be enough to determine its importance in the root library and the rationality of the match with the text. Therefore, the embodiment of the present application can further combine more factors that affect the importance of the root word on the basis of the word frequency inverse document frequency of the candidate root word, and process the word frequency inverse document frequency of the candidate root word to obtain a weight parameter that can more comprehensively and accurately reflect the importance of the root word and the degree of association with the text.

[0062] In order to better implement the method of the embodiment of the present application, in a possible example, step 202 may include: Figure 3 The following steps are shown:

[0063] Step 301, determining the document length weight of each of the documents according to the document length of each of the candidate word roots;

[0064] Step 302, determining the term weight of each candidate root word according to the part-of-speech information of each candidate root word and the position information in the document;

[0065] Step 303: Determine a weight parameter of each candidate word root according to the document length weight, term weight and word frequency inverse document frequency corresponding to each candidate word root.

[0066] Regarding document length, longer documents contain more information than shorter documents. Therefore, by considering the impact of the document length of the document where the root word is located on the importance of the root word, the document length weight is obtained, which can distinguish the difference in the impact of different roots in longer documents and shorter documents on their weight parameters.

[0067] It can also be understood that the part of speech of a root word can reflect the importance and specificity of the root word, while the position information of the root word in a document can reflect its importance in the document. Therefore, the embodiment of the present application can also combine the part of speech of the candidate root word and its position information in the document to calculate the term weight of each candidate root word, and then combine it with the document length weight, which is equivalent to considering the influence of the document of the candidate root word on the weight parameter on the one hand, and considering the influence of the candidate root word itself on the weight parameter on the other hand. It comprehensively considers the influence of factors such as document length, part of speech and position on the weight parameter of the candidate root word, and can better reflect the actual importance of the candidate root word in the root library and the rationality of the match with the text.

[0068] In some embodiments, the weight parameter of the candidate root word can be expressed as:

[0069] TF-IDF′(t,d,D)=TF(t,d)×IDF(t,D)×Norm(d,D)×Boost(t,d);

[0070] Among them, t is the candidate root, d is the document where the candidate root is located, D is the root library, Norm is the document length weight, Boost is the term weight, TF is the term frequency, and IDF is the inverse document frequency.

[0071] In order to better implement the embodiment of the present application, in a possible example, the document length weight of each document is determined by the following steps:

[0072] First, obtain the document length of each of the documents;

[0073] Then, obtaining an average document length of multiple documents in the root word library;

[0074] Finally, the document length weight of each of the documents is determined according to the document length of each of the documents and the average document length.

[0075] In some embodiments, the document length can be obtained by counting the number of roots or characters in the document. For example, the document length can be 300, indicating that there are 300 roots in the document. After determining the document length of each document in the root library, the average document length of multiple documents can be calculated by taking the average of the lengths of multiple documents. For example, the multiple documents can be all documents in the root library, or they can be multiple documents collected in the root library within a period of time. The embodiments of the present application do not limit this.

[0076] In some embodiments, the document length weight may be expressed as:

[0077] Norm(d,D) = length of document d / average document length;

[0078] It can be seen that the document length weight can reflect the importance of the document weight of the document where the candidate root is located relative to the average length. The document length weight corresponding to the candidate root in the document with a longer document length is greater than 1, and the document length weight corresponding to the candidate root in the document with a shorter document length is less than 1, indicating that the candidate root may have a higher weight parameter in the document with a longer document length.

[0079] It should be noted that the root word library maintains the information of text and root words in real time or in a short period of time. Therefore, it does not take a long time to calculate the root word library to obtain information such as the document length of each document and the average length of multiple documents. In this way, the document length of each candidate root word can be compared with the document length of multiple documents in the root word library, and then the influence of the document length of the candidate root word on the weight parameter can be determined, which helps to more accurately evaluate the importance of the candidate root word in the document.

[0080] In some embodiments, the term weight of each candidate root is determined by:

[0081] First, according to the part-of-speech information of each candidate root, the part-of-speech feature of the candidate root is determined;

[0082] Then, according to the position of each candidate root in each document, determining the position feature of the candidate root;

[0083] Finally, the term weight of each candidate root is determined according to the part-of-speech feature and position feature of each candidate root.

[0084] In some embodiments, for a root word with multiple parts of speech, the frequency of the parts of speech that appear in the document where the candidate root word is located can be counted, and the part of speech with the highest frequency can be used as the part of speech of the candidate root word. Alternatively, the part of speech of the candidate root word can be determined by combining the business scenario requirements of the model design table and considering the usage of the candidate root word in a specific model table.

[0085] In some embodiments, the part of speech of the candidate root word can be converted into a vector by encoding and word embedding to obtain the corresponding part of speech feature. For example, the verb part of speech, noun part of speech and adjective part of speech can be encoded as [1,0,0], [0,1,0], [0,0,1], or as 1, 2, 3 and other mapping values, respectively. The embodiment of the present application does not limit the method of extracting the part of speech feature. However, it should be noted that if the part of speech is mapped to different mapping values, as mentioned above, the mapping values ​​corresponding to different parts of speech can be set in combination with the business scenario requirements of the model design table and the usage of the candidate root word in the specific model table, so that the part of speech weight of the candidate root word can be more in line with the design requirements of the model table and improve the accuracy of the part of speech weight.

[0086] In addition, considering that the same candidate root word may have different meanings when it appears at different positions in a document, in some embodiments, the position of each candidate root word in the document may be obtained, for example Fig. 9 As shown, the position of the candidate root e in document A can be obtained. It can be understood that considering the influence of the position information of the candidate root in the document on its importance, the candidate root that appears in the document title, summary, key information and other positions usually has a higher importance. For another example, the candidate root appears at the beginning or end of each paragraph of the document, and may play the role of introduction or summary in the paragraph, and has a higher importance. For another example, the candidate root frequently appears in a certain position of the document to strengthen the description or create a specific context, which can also enhance the importance of the candidate root.

[0087] In some embodiments, similar to part-of-speech weights, for the same candidate root, feature encoding can be performed based on its position information at different locations in different documents, for example, the beginning position is encoded as 1, the end is 2, the middle of a paragraph is 3, etc. After determining the encoding values ​​corresponding to different position information, a weight can be assigned to each position information, and then the above encoding values ​​are weighted and summed according to the weight of the position information to obtain the position weight of the candidate root. Similarly, the mapping values ​​and encoding values ​​corresponding to different position information can be set in combination with the business scenario requirements of the model design table and the usage of the candidate root in the specific model table, so that the position weight of the candidate root can be more in line with the design requirements of the model table and improve the accuracy of the position weight.

[0088] In some embodiments, the term weight of a candidate root word can be expressed as:

[0089] Boost(t,d,D)=position weight(t,d)×part-of-speech weight(t,d);

[0090] It can be understood that the word weight and position weight of the candidate root can be used to more accurately determine the term weight of the candidate root, thereby increasing the influence of the candidate root's own information on the root's importance, so as to play an important role in the root determination operation of the text and improve the efficiency and accuracy of root determination.

[0091] Step 203: determining a target root word corresponding to the text from the candidate root words according to the weight parameters of the candidate root words.

[0092] In some embodiments, the corresponding weight parameters can be calculated according to the inverse document frequency, document length weight and term weight of each candidate root using the weight parameter calculation formula, and multiple candidate roots can be sorted according to the numerical values ​​of the weight parameters to obtain sorting results, and a certain number of candidate roots can be selected from the sorting results of the multiple candidate roots to finally determine a target root that best matches the text.

[0093] See also Figure 8 , taking the text "username" as an example, after calculating the weight parameters of multiple candidate root words, the three candidate root words with the largest weight parameters can be determined from the sorting results. For example, the weight parameter of "user_name" is 1, the weight parameter of "username" is 0.8, and the weight parameter of "usename" is 0.5. Finally, "user_name" can be determined as the target root word that matches the text, and the second and third ranked candidate root words are used as reference roots for subsequent model table design.

[0094] In order to better implement the embodiment of the present application, in a possible example, step 203 may include the following steps:

[0095] First, the candidate word roots are sorted according to the weight parameter of each candidate word root to obtain a sorting result, and a first candidate word root and a second candidate word root are determined according to the sorting result;

[0096] Finally, when the difference between the first weight parameter of the first candidate root and the second weight parameter of the second candidate root is greater than a preset threshold, the first candidate root is determined as the target root.

[0097] The first candidate root word refers to the candidate root word with the largest value of the weight parameter in the sorting result, and the second candidate root word refers to the candidate root word with the second largest value of the weight parameter in the sorting result.

[0098] In some embodiments, the embodiments of the present application can first determine the first candidate root and the second candidate root from the sorting results of multiple candidate roots, and then compare the difference between the first weight parameter and the second weight parameter with the size of the preset threshold. It can be understood that the weight parameter of the second candidate root is greater than the other weight parameters except the first candidate root. Therefore, if the difference between the first weight parameter and the second weight parameter is greater than the preset threshold, it means that the first weight parameter is significantly greater than the second weight parameter, which is also greater than the weight parameters of the other multiple candidate roots, indicating that the weight parameter of the first candidate root is not only the highest, but also significantly higher than the weight parameters of all other candidate roots. Therefore, it can be directly determined that the first candidate root is the target root that best matches the text.

[0099] In order to better implement the embodiments of the present application, in a possible example, the method of the present application further includes:

[0100] First, when the difference between the first weight parameter and the second weight parameter is less than or equal to a preset threshold, a target number of third candidate word roots are determined according to the sorting result;

[0101] Then, calculating the matching degree between the first candidate root, the second candidate root and the third candidate root and the text respectively;

[0102] Finally, based on the matching degree, the target root word is determined from the first candidate root word, the second candidate root word and the third candidate root word.

[0103] It can be understood that when the difference between the first weight parameter and the second weight parameter is less than or equal to the preset threshold, it means that the weight parameters of the first candidate root and the second candidate root are relatively close. In order to ensure the accuracy of the root determination result, the first candidate root may not be directly determined as the target root, but the target number of third candidate roots may be further determined based on the sorting result. For example, the target number may be 1 or 2, or the difference between the first weight parameter and the weight parameters of other candidate roots may be calculated, and the number of candidate roots whose difference is less than or equal to the preset threshold may be used as the target number. In this way, multiple third candidate roots with values ​​close to the first weight parameter may be screened out first, and then a unique target root may be determined from the first candidate root, the second candidate root and the third candidate root.

[0104] In some embodiments, the degree of match between the candidate root and the text can be determined from the perspective of the design requirements and application scenarios of the model table. Taking the model table of the user's online shopping data as an example, assuming that the text is "reliability of product quality", the corresponding two candidate root words are "reliability" and "stability". The design requirements and application scenarios of the model table of the user's online shopping data can be analyzed. Considering that the meaning of "stability" is more inclined to the stability and smoothness of the product during operation, and "reliability" emphasizes the effectiveness of the use of the product, even if "reliability" is the second or third candidate root, it can be determined as the target root.

[0105] It should be noted that when the above situation occurs where it is necessary to determine a unique target root from the first candidate root, the second candidate root and the third candidate root, a notification can also be sent to the model table designer to obtain the candidate root finally confirmed by the model table designer as the target root. The embodiment of the present application does not limit this.

[0106] In order to better implement the embodiments of the present application, in a possible example, the method of the present application further includes:

[0107] First, scan and parse the data source of word roots to obtain new word roots that are not stored in the word root library;

[0108] Then, the new word roots are subjected to deduplication processing and correction processing to obtain processed new word roots;

[0109] Finally, the processed new word roots are added to the word root library.

[0110] In some embodiments, in order to ensure the comprehensiveness and accuracy of the root word library, the root word library may be continuously checked and maintained. For example, the root word library may be checked for obvious errors and duplicate root words, but it should be noted that for relatively common root words such as "user_name" and "username", the two may have different meanings in different business scenarios, so they are not considered duplicate root words in the root word library.

[0111] In some embodiments, the data source may be scanned periodically. When a new root word that is not stored is found, the new root word may be preprocessed to remove duplications and correct it. After the preprocessing is completed, it may be automatically added to the root word library. For example, when two identical new roots are scanned at the same time, only one of the new roots may be retained. For example, when an obviously erroneous new root word such as "userr" is scanned, it may be corrected.

[0112] It should be noted that the above scanning operation can be performed during the off-peak period of business to ensure stable operation of the system. In addition, for special scenarios, such as when the data source is not a database but a JSON string parsing data lake warehouse, or a log data lake warehouse, the embodiment of the present application can provide a variety of root import solutions. For example, in addition to importing new roots through automatic scanning, it can also support the import of new root data into the root library through Excel batch import.

[0113] Through the above method, the embodiment of the present application can realize the automatic maintenance of the root word library, and can continuously improve the root word library, making the root word data more accurate and rich, and providing a more accurate basis for subsequent text processing and analysis.

[0114] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a text root determination device for implementing the text root determination method involved in the above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more text root determination devices provided below can refer to the limitations of the text root determination method above, and the specific limitations will not be repeated here.

[0116] The text root determination device provided in this embodiment can be integrated into a terminal device. Fig.10 As shown, the text root determination device may include: a candidate root search module 401 , a weight parameter determination module 402 , and a target root determination module 403 .

[0117] The root determination device of the above-mentioned text first searches for the candidate root corresponding to the text from the root library through the candidate root search module 401, and then determines the weight parameter of the candidate root according to the word frequency inverse document frequency, part of speech information, document information of the document where the candidate root is located, and the position information of the candidate root in the document through the weight parameter determination module 402. Finally, the target root determination module 403 determines the target root corresponding to the text from the candidate root according to the weight parameter of the candidate root, so that the self information of each candidate root can be combined with the document information where it is located, and the rational weight of each candidate root in the root determination of the text can be determined more comprehensively and accurately on the basis of the word frequency inverse document frequency, thereby improving the accuracy of the root determination of the text. In this way, a most accurate and non-repetitive target root can be determined for different texts to ensure the uniformity of matching roots, thereby also improving the accuracy and standardization of the content of the generated data table.

[0118] In some embodiments, the weight parameter determination module 402 is further specifically configured to:

[0119] Determining a document length weight of each of the documents according to the document length of each of the candidate word roots;

[0120] Determining a term weight of each candidate root word according to the part-of-speech information of each candidate root word and the position information in the document;

[0121] The weight parameter of each candidate word root is determined according to the document length weight, term weight and word frequency inverse document frequency corresponding to each candidate word root.

[0122] In some embodiments, the weight parameter determination module 402 is further specifically configured to:

[0123] Obtaining the document length of each of the documents;

[0124] Obtaining an average document length of multiple documents in the root word library;

[0125] The document length weight of each of the documents is determined according to the document length of each of the documents and the average document length.

[0126] In some embodiments, the weight parameter determination module 402 is further specifically configured to:

[0127] Determine the part-of-speech feature of each candidate root word according to the part-of-speech information of each candidate root word;

[0128] Determine the position feature of each candidate word root according to the position of each candidate word root in each document;

[0129] The term weight of each candidate word root is determined according to the part-of-speech feature and position feature of each candidate word root.

[0130] In some embodiments, the target root determination module 403 is specifically used to:

[0131] Sorting the candidate word roots according to the weight parameter of each candidate word root to obtain a sorting result, and determining a first candidate word root and a second candidate word root according to the sorting result;

[0132] When the difference between the first weight parameter of the first candidate root and the second weight parameter of the second candidate root is greater than a preset threshold, the first candidate root is determined as the target root.

[0133] In some embodiments, the text root determination device further includes:

[0134] A third candidate root determination module, configured to determine a target number of third candidate root words according to the sorting result when the difference between the first weight parameter and the second weight parameter is less than or equal to a preset threshold;

[0135] A matching degree determination module, used for calculating the matching degree between the first candidate root, the second candidate root and the third candidate root and the text respectively;

[0136] A target root word determination module is used to determine the target root word from the first candidate root word, the second candidate root word and the third candidate root word based on the matching degree.

[0137] In some embodiments, the text root determination device further includes:

[0138] A new root word acquisition module, used to scan and parse the data source of root words to obtain new root words that are not stored in the root word library;

[0139] A new root word processing module, used for performing deduplication processing and correction processing on the new root word to obtain a processed new root word;

[0140] The root word library maintenance module is used to add the processed new root word to the root word library.

[0141] Based on the same inventive concept, the embodiment of the present application further provides a computer device, which may be a server or a terminal device, and includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned text root determination method when executing the computer program. Thereby, various functions are realized, for example:

[0142] Search for candidate word roots corresponding to the text from the word root library;

[0143] Determine a weight parameter of the candidate root according to the inverse document frequency of the candidate root, part of speech information, document information of the document where the candidate root is located, and position information of the candidate root in the document;

[0144] According to the weight parameters of the candidate word roots, a target word root corresponding to the text is determined from the candidate word roots.

[0145] Adopting the scheme of the embodiment of the present application, the candidate root corresponding to the text is searched from the root library, and the weight parameter of the candidate root is determined according to the word frequency inverse document frequency, part of speech information, document information of the document where the candidate root is located, and the position information of the candidate root in the document, so that the self information of each candidate root can be considered in combination with the document information where it is located, and the rational weight of each candidate root in the root determination of the text can be determined more comprehensively and accurately on the basis of the word frequency inverse document frequency, thereby improving the accuracy of the root determination of the text. Finally, according to the weight parameter of the candidate root, the target root corresponding to the text is determined from the candidate root. In this way, a most accurate and non-repetitive target root can be determined for the roots of different texts, ensuring the uniformity of the matching roots, thereby also improving the accuracy and standardization of the generated data table content.

[0146] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0147] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows: Fig.11As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for determining the root of a text is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0148] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0149] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0150] Since the computer program stored in the computer-readable storage medium can execute the root determination method of any text provided in the embodiments of the present application, the beneficial effects that can be achieved by the root determination method of any text provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0151] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.

[0152] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0153] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0154] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0155] In the above-mentioned text root determination device, computer-readable storage medium, computer device, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the above-described text root determination device, computer-readable storage medium, computer program product, computer device, and the specific working process and beneficial effects that can be brought about can refer to the description of the text root determination method in the above embodiment, and the specific details will not be repeated here.

[0156] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above is a detailed introduction to the root determination method, device, computer equipment, computer-readable storage medium and computer program product of a text provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for determining the root of a text, It is characterized in that The method comprises: Search for candidate word roots corresponding to the text from the word root library; Determine a weight parameter of the candidate root according to the inverse document frequency of the candidate root, part of speech information, document information of the document where the candidate root is located, and position information of the candidate root in the document; According to the weight parameters of the candidate word roots, a target word root corresponding to the text is determined from the candidate word roots.

2. The method according to claim 1, It is characterized in that The step of determining the weight parameter of the candidate root according to the word frequency inverse document frequency, part of speech information, document information of the document where the candidate root is located, and position information of the candidate root in the document includes: Determining a document length weight of each of the documents according to the document length of each of the candidate word roots; Determining a term weight of each candidate root word according to the part-of-speech information of each candidate root word and the position information in the document; The weight parameter of each candidate word root is determined according to the document length weight, term weight and word frequency inverse document frequency corresponding to each candidate word root.

3. The method according to claim 2, It is characterized in that The step of determining the document length weight of each document according to the document length of each candidate word root, comprises: Obtaining the document length of each of the documents; Obtaining an average document length of multiple documents in the root word library; The document length weight of each of the documents is determined according to the document length of each of the documents and the average document length.

4. The method according to claim 2, It is characterized in that Determining the term weight of each candidate root word according to the part-of-speech information of each candidate root word and the position information in the document includes: Determine the part-of-speech feature of each candidate root word according to the part-of-speech information of each candidate root word; Determine the position feature of each candidate word root according to the position of each candidate word root in each document; The term weight of each candidate word root is determined according to the part-of-speech feature and position feature of each candidate word root.

5. The method according to claim 1, It is characterized in that The step of determining a target root word corresponding to the text from the candidate root words according to the weight parameters of the candidate root words comprises: Sorting the candidate word roots according to the weight parameter of each candidate word root to obtain a sorting result, and determining a first candidate word root and a second candidate word root according to the sorting result; When the difference between the first weight parameter of the first candidate root and the second weight parameter of the second candidate root is greater than a preset threshold, the first candidate root is determined as the target root.

6. The method according to claim 5, It is characterized in that The method further comprises: When the difference between the first weight parameter and the second weight parameter is less than or equal to a preset threshold, determining a target number of third candidate word roots according to the sorting result; Calculating the matching degree between the first candidate root, the second candidate root and the third candidate root and the text respectively; Based on the matching degree, the target root word is determined from the first candidate root word, the second candidate root word, and the third candidate root word.

7. The method according to claim 1, It is characterized in that The method comprises: Scan and parse the data source of word roots to obtain new word roots that are not stored in the word root library; Performing deduplication processing and correction processing on the new word roots to obtain processed new word roots; The processed new word root is added to the word root library.

8. A device for determining a root of a text, It is characterized in that The device comprises: A candidate root word search module is used to search for candidate root words corresponding to a text from a root word library; A weight parameter determination module, used to determine the weight parameter of the candidate root according to the word frequency inverse document frequency, part of speech information, document information of the document where the candidate root is located, and position information of the candidate root in the document; The target root word determination module is used to determine the target root word corresponding to the text from the candidate root words according to the weight parameters of the candidate root words.

9. A computer device, It is characterized in that The invention comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the method for determining a root of a text as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the method for determining a root of a text according to any one of claims 1 to 7.

11. A computer software, It is characterized in that The computer software includes a computer program, and the computer program is used by a processor to execute the steps of the text root determination method as described in any one of claims 1 to 7.

Images