Data processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a computer readable storage medium. The method comprises the steps that to-be-processed words, a formal dictionary and a candidate dictionary are obtained, the formal dictionary comprises at least one formal synonym group, the candidate dictionary comprises at least one candidate synonym group, and the formal synonym group and the candidate synonym group are both sets of synonyms; determining a first matching degree between the to-be-processed word and each candidate synonym group; when any first matching degree is larger than a first matching degree threshold value, the to-be-processed word is written into a first synonym group, an evaluation index of the first synonym group is determined, the first synonym group is a candidate synonym group corresponding to any first matching degree, and the evaluation index is used for representing the probability that the first synonym group is added into a formal dictionary; and moving the first synonym group from the candidate dictionary to a formal dictionary based on the evaluation index.

Description

Technical Field

[0001] The present application relates to natural language processing technology, and in particular to a data processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] Natural Language Processing (NLP) is a branch of computer science, artificial intelligence and linguistics that aims to enable computers to understand and process human language. In the field of natural language processing, a synonym domain dictionary is a word resource that focuses on a specific field. It not only contains standard terms in the field, but also includes other words with the same or very similar meanings as these terms, thus forming a synonym set, i.e., a synonym group. However, in the relevant technology, there is a lack of effective management and continuous updating mechanism for synonym groups in the synonym domain dictionary, resulting in poor accuracy and real-time performance of the synonym domain dictionary, thus affecting its effect in practical applications. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, device, electronic device and computer-readable storage medium, which can realize dynamic update of synonym groups from candidate dictionaries to formal dictionaries.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a data processing method, the method comprising:

[0006] Obtain a word to be processed, a formal dictionary, and a candidate dictionary, wherein the formal dictionary includes at least one formal synonym group, the candidate dictionary includes at least one candidate synonym group, and both the formal synonym group and the candidate synonym group are sets of synonyms;

[0007] Determining a first matching degree between the word to be processed and each of the candidate synonym groups;

[0008] When any of the first matching degrees is greater than a first matching degree threshold, the word to be processed is written into a first synonym group, and an evaluation index of the first synonym group is determined, wherein the first synonym group is a candidate synonym group corresponding to any of the first matching degrees, and the evaluation index is used to represent the probability of the first synonym group being added to the formal dictionary;

[0009] Based on the evaluation index, the first synonym group is moved from the candidate dictionary to the official dictionary.

[0010] The present application provides a data processing device, the device comprising:

[0011] A data acquisition module, used to acquire a to-be-processed word, a formal dictionary, and a candidate dictionary, wherein the formal dictionary includes at least one formal synonym group, the candidate dictionary includes at least one candidate synonym group, and both the formal synonym group and the candidate synonym group are sets of synonyms;

[0012] A matching degree determination module, used to determine a first matching degree between the word to be processed and each of the candidate synonym groups;

[0013] a data writing module, configured to write the word to be processed into a first synonym group when any of the first matching degrees is greater than a first matching degree threshold, and determine an evaluation index for the first synonym group, wherein the first synonym group is a candidate synonym group corresponding to any of the first matching degrees, and the evaluation index is used to represent the probability of the first synonym group being added to the formal dictionary;

[0014] A data transmission module is used to move the first synonym group from the candidate dictionary to the formal dictionary based on the evaluation index.

[0015] A memory for storing computer executable instructions or computer programs;

[0016] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0017] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in the embodiment of the present application when executed by a processor.

[0018] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the data processing method provided in the embodiment of the present application is implemented.

[0019] The embodiments of the present application have the following beneficial effects:

[0020] The first matching degree between the word to be processed and each candidate synonym group is determined. Only when any first matching degree is greater than the first matching degree threshold, the word to be processed is written into the first synonym group, thereby reducing the misclassification of the word to be processed and ensuring the accuracy and effectiveness of the synonym group. At the same time, the evaluation index of the first synonym group is updated. Based on the evaluation index, the first synonym group is moved from the candidate dictionary to the formal dictionary. Through the dynamic update of the synonym group from the candidate dictionary to the formal dictionary, the accuracy and real-time performance of the formal dictionary are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a structural diagram of the data processing system architecture provided in an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of the structure of an electronic device for data processing provided in an embodiment of the present application;

[0023] Figure 3A It is a first flow chart of the data processing method provided in the embodiment of the present application;

[0024] Figure 3B is a second flow chart of the data processing method provided in an embodiment of the present application;

[0025] Figure 3C is a third flow chart of the data processing method provided in the embodiment of the present application;

[0026] Figure 3D is a fourth flow chart of the data processing method provided in an embodiment of the present application;

[0027] Figure 3E is a fifth flow chart of the data processing method provided in the embodiment of the present application;

[0028] Figure 3F is a sixth flow chart of the data processing method provided in an embodiment of the present application;

[0029] Figure 3G is a seventh flow chart of the data processing method provided in the embodiment of the present application;

[0030] Figure 4 is a first principle diagram of a data processing method provided in an embodiment of the present application;

[0031] Figure 5 is a second principle diagram of the data processing method provided in an embodiment of the present application;

[0032] Figure 6 is a third principle diagram of the data processing method provided in an embodiment of the present application;

[0033] Figure 7 is a fourth principle diagram of the data processing method provided in an embodiment of the present application;

[0034] Figure 8 is a fifth principle diagram of the data processing method provided in an embodiment of the present application;

[0035] Fig. 9 This is the sixth principle diagram of the data processing method provided in the embodiment of the present application.

[0036] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0041] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0042] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0043] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0044] 1) Dictionary (field dictionary): A dictionary compiled for a specific discipline, industry or field, focusing on the terms and concepts in a specific field, providing accurate and professional language references for professionals and people interested in this field. In the embodiment of the present application, the dictionary includes a formal dictionary and a candidate dictionary. The formal dictionary is used to store confirmed synonym groups, and the candidate dictionary is used to store synonym groups to be evaluated.

[0045] 2) Nested Synonyms: A structure in which words with similar meanings are grouped and nested in a document. In this structure, words with similar meanings are grouped into a synonym group. The nested synonym structure can reflect the semantic association between words and allow more refined semantic queries and text analysis.

[0046] 3) Synonym group: A synonym group is a collection of synonymous words. For example, a synonym group can refer to a group of words that are semantically interchangeable. These words have the same or very similar meanings, but there may be subtle differences in their use, such as in context, tone, formality, collocation, etc.

[0047] 4) Embedding: It is a core technology in the field of natural language processing. Its main purpose is to map words, phrases or other language units in the text into a continuous vector space. Each unit (such as a word) is represented by a vector of fixed length in this space. This vector is called an embedding vector, which can capture and represent the semantic information and contextual information of the original language unit.

[0048] 5) Cosine Similarity: It is a commonly used similarity measurement method used to measure the degree of similarity between two vectors in direction. Cosine similarity calculates the similarity based on the angle between the vectors. The value range is -1 to 1. For example, when the directions of the two vectors are exactly the same, the cosine similarity is 1, indicating that the two vectors are very similar; when the directions of the two vectors are completely opposite, the cosine similarity is -1, indicating that the two vectors are very dissimilar; when the two vectors are orthogonal (i.e., perpendicular to each other), the cosine similarity is 0.

[0049] 6) Base Word: In linguistics and natural language processing, a base word refers to a word that serves as the core or reference standard in a group of words. A base word has clear semantics, is frequently used, and is at the center of the vector space of a synonym group. The base word can represent or reflect the semantic characteristics of the entire synonym group, thereby making the synonym group itself more representative.

[0050] 7) Large Language Models (LLM): refers to pre-trained language models with a very large number of parameters. The number of parameters of large language models usually reaches billions or even hundreds of billions, which enables them to capture complex patterns and subtle differences in language. They can understand and generate human language through unsupervised learning on large text corpora.

[0051] 8) Prompt: refers to the input text used to guide the model to generate text or perform specific tasks. The prompt usually contains some information about the expected output. It can help the model understand the context and, in some cases, guide the model to generate a response that meets specific requirements.

[0052] 9) Stop words: In natural language processing, stop words are some common words that are pre-defined and usually automatically filtered out during text analysis and processing. They include some functional words, such as prepositions, articles, and conjunctions, which appear frequently in the text but are not directly related to expressing a specific meaning or theme.

[0053] The embodiments of the present application provide a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can realize dynamic updating of synonym groups from a candidate dictionary to a formal dictionary.

[0054] The data processing method provided in the embodiment of the present application can be implemented by the terminal or the server alone, or can be implemented by the terminal and the server together. For example, the terminal alone undertakes the data processing method described below, or the terminal sends a data processing request containing the words to be processed to the server, and the server executes the data processing method based on the received data processing request containing the words to be processed.

[0055] The electronic device for data processing provided in the embodiments of the present application may be various types of terminals or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0056] See also Figure 1 , Figure 1 It is a structural diagram of the data processing system architecture provided in the embodiment of the present application. In the data processing system 10 provided in the embodiment of the present application, in order to support a data processing application, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0057] Terminal 400 may be used to obtain a data processing request including a word to be processed.

[0058] In some embodiments, a data processing plug-in may be implanted in the client running in the terminal 400 to implement the data processing method locally on the client. For example, the terminal 400 calls the data processing plug-in to implement the data processing method, obtains the formal dictionary and the candidate dictionary, and determines the first matching degree between the word to be processed and each candidate synonym group in the candidate dictionary; when any first matching degree is greater than the first matching degree threshold, the word to be processed is written into the first synonym group, and the evaluation index of the first synonym group is updated; based on the evaluation index, the first synonym group is moved from the candidate dictionary to the formal dictionary.

[0059] In some embodiments, after the terminal 400 obtains a data processing request containing a word to be processed, the data processing interface of the server 200 (which can be provided in the form of a cloud service) is called, and the server 200 implements a data processing method through a data processing plug-in, obtains a formal dictionary and a candidate dictionary, and determines the first matching degree between the word to be processed and each candidate synonym group in the candidate dictionary; when any first matching degree is greater than a first matching degree threshold, the word to be processed is written into the first synonym group, and the evaluation index of the first synonym group is updated; based on the evaluation index, the first synonym group is moved from the candidate dictionary to the formal dictionary, and the formal dictionary is returned to the terminal 400, so that the formal dictionary can be used in the terminal 400.

[0060] The data processing method of the embodiment of the present application can be applied to various scenarios, for example, application fields, natural science fields and literary and artistic fields. In the field of natural sciences (for example, physics, chemistry, geography and astronomy, etc.), after the terminal 400 obtains the original expected data containing professional words, word usage, word semantics and word context in the field of natural sciences, the terminal 400 constructs a data processing request containing the word to be processed based on the received original expected data, and the terminal 400 calls the data processing interface of the server 200. The server 200 implements the data processing method through the data processing plug-in, obtains the natural science field dictionary and the natural science field candidate dictionary, and determines the first matching degree between the word to be processed and each candidate synonym group in the natural science field candidate dictionary; when any first matching degree is greater than the first matching degree threshold, the word to be processed is written into the first synonym group, and the evaluation index of the first synonym group is updated; based on the evaluation index, the first synonym group is moved from the natural science field candidate dictionary to the natural science field dictionary, and the natural science field dictionary is returned to the terminal 400, so as to use the natural science field dictionary in the terminal 400.

[0061] In an application field (for example, a mobile application field, a computer application field, and a cloud application field, etc.), after the terminal 400 obtains the community data generated by the application in real time (for example, comment data in a user forum), the terminal 400 constructs a data processing request containing a word to be processed based on the received community data, and the terminal 400 calls the data processing interface of the server 200. The server 200 implements a data processing method through a data processing plug-in, obtains an application field dictionary and an application field candidate dictionary, and determines a first matching degree between the word to be processed and each candidate synonym group in the application field candidate dictionary; when any first matching degree is greater than a first matching degree threshold, the word to be processed is written into the first synonym group, and an evaluation index of the first synonym group is updated; based on the evaluation index, the first synonym group is moved from the application field candidate dictionary to the application field dictionary, and the application field dictionary is returned to the terminal 400, so that the application field dictionary can be used in the terminal 400.

[0062] In an actual application scenario, the relevance of user comments in the application can be judged through the application domain dictionary, and the topics of the comment data in the user forum of the application deployed on the terminal 400 can be extracted through natural language processing methods, and the topics of the comment data can be matched with the application domain dictionary to judge the relevance of the comment data to the current application. According to the relevance judgment results "whether it is relevant, the hit synonym group, etc.", relevance tags are added to the comment data, and the comment data with the relevance tags are stored in the comment database for standby use.

[0063] In another actual application scenario, the hot comments in the application can be subject-mined through the application domain dictionary. Terminal 400 obtains comment data with relevant tags added from the comment database, and builds a hot topic list based on the number of times the topic of the comment data hits each synonym group in the application domain dictionary, thereby helping the application to promptly understand the focus of user attention and provide data support for product optimization and operational decisions.

[0064] In another actual application scenario, data retrieval can be performed on topic-based comments in the application through the application domain dictionary. Terminal 400 obtains comment data with relevance tags added from the comment database, and classifies the comment data with relevance tags added based on the topic of the comment data to implement the function of reverse query of comment data by topic. For example, when comment data related to topic A is clicked, terminal 400 can return comment data related to topic A.

[0065] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device for data processing provided in an embodiment of the present application, Figure 2 The electronic device 500 shown may be Figure 1The terminal 400 or the server 200 in the electronic device 500 includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 540 .

[0066] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, a hard disk drive, an optical disk drive, etc. The memory 550 may optionally include one or more storage devices that are physically located away from the processor 510.

[0068] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0069] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0070] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0071] A network communication module 552, for reaching other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0072] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The data processing device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a data acquisition module 5551, a matching degree determination module 5552, a data writing module 5553 and a data transmission module 5554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0073] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0074] As mentioned above, the electronic device implementing the data processing method of the embodiment of the present application may be a terminal, a server, or a combination of the two, so the execution subject of each step will not be repeatedly described below, see Figure 3A , Figure 3A is a first flow chart of the data processing method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0075] In step 101, a word to be processed, a formal dictionary, and a candidate dictionary are obtained.

[0076] As an example, the word to be processed refers to a word that can express a certain semantics alone. The formal dictionary may include at least one formal synonym group, and the candidate dictionary may include at least one candidate synonym group. Both the formal synonym group and the candidate synonym group are sets of synonyms. For the initialized formal dictionary and candidate dictionary, the formal dictionary and the candidate dictionary may include empty phrases.

[0077] The formal dictionary is used to store synonym groups that meet the evaluation indicators, and the candidate dictionary is used to store synonym groups that do not meet the evaluation indicators. Synonym groups refer to a collection of words with the same or very similar meanings in a specific language.

[0078] For example, in the application scenario of the mobile application field dictionary (APP field dictionary), dictionary 1 is [{"communication tool", "instant messaging"}, {"social media", "social platform"}, {"post", "post publishing", "post sharing"}], among which the two words "communication tool" and "instant messaging" have very similar meanings, that is, the word set {"communication tool", "instant messaging"} is a synonym group, among which the mobile application field dictionary refers to a dictionary built based on a specific mobile application, and the mobile application field dictionary contains terms and words related to the development, design, operation and maintenance of mobile applications.

[0079] The formal dictionary and candidate dictionary in the embodiment of the present application provide a structured framework for the management of synonym groups, which helps to realize the dynamic update and management of synonym groups and improve the accuracy and real-time performance of the formal dictionary.

[0080] In some embodiments, Figure 3A The "obtaining the words to be processed" in the shown step 101 can be implemented by the following steps: obtaining text information, and performing word segmentation processing on the text information to obtain at least one word segmentation; performing stop word removal processing on at least one word segmentation to obtain the removed word segmentation; based on the frequency of occurrence of the removed word segmentation in the text information, determining the words to be processed from the removed word segmentations.

[0081] As an example, text information refers to all original expected data that can provide professional terms, word usage, word semantics and word context in a specific field. The text information is segmented to obtain at least one segmented word, and stop word removal is performed on at least one segmented word to obtain the segmented word after removal. Based on the frequency of occurrence of the segmented word after removal in the text information, the segmented word with the highest frequency of occurrence is determined as the word to be processed, or, a frequency threshold is set to determine the segmented word with an occurrence frequency greater than the frequency threshold as the word to be processed.

[0082] For example, in the application scenario of a dictionary in the field of mobile applications, text information may include relevant documents of mobile applications, community data, and professional terminology sets, etc. Among them, relevant documents of mobile applications include description information of mobile applications in application stores, high-quality user reviews, and application description documents, etc., community data include posts in internal discussion forums of applications, etc., and professional terminology sets include professional knowledge and industry terminology in the corresponding fields.

[0083] For the Chinese text information "I like swimming, swimming makes me healthy, and swimming is an interesting sport." Based on the Chinese word segmentation tool, the input text is segmented to obtain a word set ["I", "like", "swimming", "swimming", "make", "I", "healthy", "swimming", "is", "a kind of", "interesting", "of", "sport"], and the stop words "I", "make", "is", "a kind of" and "of" in the word segmentation set are removed to obtain the removed word segmentations ["like", "swimming", "swimming", "healthy", "swimming", "interesting", "sport"], among which the occurrence frequencies of "like", "healthy", "interesting" and "sports" are all once, and the occurrence frequency of "swimming" is three times, and "swimming" with the highest occurrence frequency is determined as the word to be processed.

[0084] In the embodiment of the present application, by performing word segmentation and stop word removal processing on the text information, the noise in the text information (such as punctuation marks, common meaningless words, etc.) is removed, thereby making the words to be processed in the text information more prominent, helping to focus on the words that truly convey meaning, thereby improving the efficiency of text analysis.

[0085] In step 102, a first matching degree between the word to be processed and each candidate synonym group is determined.

[0086] As an example, based on the semantic similarity between the word to be processed and the benchmark word of each candidate synonym group, the first matching degree between the word to be processed and each candidate synonym group is determined, wherein each synonym group corresponds to a unique benchmark word, and the benchmark word is the word that best represents the semantic characteristics of the synonym group.

[0087] In some embodiments, Figure 3A The illustrated step 102 may be implemented by the following steps: text encoding the word to be processed to obtain a first encoding feature of the word to be processed; text encoding the benchmark word of each candidate synonym group to obtain a second encoding feature of each benchmark word; and using the similarity between the first encoding feature and the second encoding feature as a first matching degree.

[0088] As an example, the trained text encoder is used to perform text encoding processing on the word to be processed and the benchmark word of each candidate synonym group, respectively, to obtain the first encoding feature of the word to be processed and the second encoding feature of each benchmark word. The text encoder may include a word vector model (Word2Vec model), a global word vector model (GloVe model), and a bidirectional encoder (BERT, Bidirectional Encoder Representations from Transformers), or a large text embedding coding model, such as a Hunyuan Embedding model and a large embedding model (GBELarge Embedding), etc., which are not limited here.

[0089] The method for determining the similarity between the first coding feature and the second coding feature includes, but is not limited to, calculating the Euclidean distance, Mahalanobis distance, Pearson correlation coefficient or cosine similarity between the first coding feature and the second coding feature. Taking cosine similarity as an example, the cosine similarity between the first coding feature of the word to be processed and the second coding feature of the benchmark word of each candidate synonym group is determined as the first matching degree between the word to be processed and each candidate synonym group.

[0090] In the embodiment of the present application, the cosine similarity between the first coding feature and the second coding feature is essentially a comparison of the vector directions of the first coding feature and the second coding feature. Due to its insensitivity to the vector length, when the word frequency fluctuates, the semantic similarity between the first coding feature and the second coding feature can still be stably evaluated, thereby accurately capturing the deep semantic relationship between the word to be processed and the benchmark word; in addition, in the field of natural language processing, cosine similarity has demonstrated excellent performance in multiple tasks such as document classification, information retrieval, and word sense disambiguation, and in terms of computational efficiency, cosine similarity has obvious advantages over Euclidean distance and Mahalanobis distance, and is particularly suitable for processing large-scale data sets.

[0091] In some embodiments, before "text encoding the baseline word of each candidate synonym group to obtain the second encoding feature of each baseline word", the following processing can also be performed on each candidate synonym group: determine the number of first words included in the candidate synonym group; based on the number of first words, determine the baseline word of the candidate synonym group.

[0092] As an example, based on the number of first words in the candidate synonym group, a corresponding reference word determination strategy is selected, and based on the corresponding reference word determination strategy, a reference word of the candidate synonym group is determined.

[0093] In the embodiment of the present application, the benchmark word is selected according to the number of the first words in the synonym group, which can achieve personalized customization of the selection of the benchmark word for the synonym group, ensuring that the selection of the benchmark word is not only more in line with specific semantic needs, but also more in line with the specific contextual background, and at the same time can prevent the problem of overfitting in the process of dictionary expansion.

[0094] In some embodiments, see Figure 3B , Figure 3B This is a second flow chart of the data processing method provided in an embodiment of the present application. "Determining the benchmark word of the candidate synonym group based on the number of the first word" can be achieved by following steps 201 to 203, which are described in detail below.

[0095] In step 201, when the number of the first word is equal to 1, the first word is used as a base word of the candidate synonym group.

[0096] As an example, when the number of first words is equal to 1, the benchmark word determination strategy is: use the only first word as the benchmark word of the candidate synonym group.

[0097] For example, for the candidate synonym group {"happy"}, the candidate synonym group only includes one first word "happy", that is, the number of first words in the candidate synonym group is equal to 1, and the first word "happy" is used as the base word of the candidate synonym group.

[0098] In step 202, when the number of first words is equal to 2, the first word with the least number of characters is determined as the base word of the candidate synonym group.

[0099] As an example, when the number of first words is equal to 2, the benchmark word determination strategy is: when the number of characters of the first words is not equal, the first word with the least number of characters is determined as the benchmark word of the candidate synonym group; when the number of characters of the first words is equal, a first word is randomly selected as the benchmark word of the candidate synonym group.

[0100] For example, for the candidate synonym group {"happy", "heart is full of joy"}, the candidate synonym group includes two first words "happy" and "heart is full of joy", that is, the number of first words in the candidate synonym group is equal to 2, the number of characters of the first word "happy" is 2, and the number of characters of the first word "heart is full of joy" is 4, and the first word "happy" with the least number of characters is determined as the benchmark word of the candidate synonym group.

[0101] In the embodiment of the present application, the base word is directly determined from the synonym group by the number of characters of the words in the synonym group, thereby reducing unnecessary calculations and improving processing efficiency.

[0102] In step 203, when the number of first words is greater than 2, a reference word is selected from the first words included in the candidate synonym group.

[0103] As an example, when the number of first words is greater than 2, the benchmark word determination strategy is: select the word that best represents the semantic characteristics of the synonym group from the first words included in the candidate synonym group, and use the word that best represents the semantic characteristics of the synonym group as the benchmark word of the candidate synonym group.

[0104] In the embodiment of the present application, selecting the word that best represents the semantic characteristics of the synonym group as the reference word helps to more accurately reflect the semantic center of the synonym group, thereby improving the matching accuracy of the synonym group with other words or other synonym groups.

[0105] In some embodiments, see Figure 3C , Figure 3C is a third flow chart of the data processing method provided in the embodiment of the present application, Figure 3B The step 203 of "selecting a reference word from the first word included in the candidate synonym group" shown in the figure can be implemented by executing the following steps 2031A to 2033A for any first word in the candidate synonym group, which is described in detail below.

[0106] In step 2031A, a second similarity between the first word and other words is determined.

[0107] As an example, text encoding processing is performed on each first word in the candidate synonym group to obtain a text encoding feature of each first word, and a second similarity between the text encoding feature of the first word and the text encoding features of other words is determined, wherein the other words are words other than the first word in the candidate synonym group. The specific steps of the text encoding processing refer to the method of performing text encoding processing on the processed words above, and the specific steps of determining the second similarity refer to the method of determining the similarity between the first encoding feature and the second encoding feature above, which will not be repeated here.

[0108] For example, for the candidate synonym group {"happy", "delighted", "heart is blossoming"}, text encoding processing is performed on each first word in the candidate synonym group, and text encoding features of "happy", "delighted", and "heart is blossoming" are obtained respectively, and the second similarity between the text encoding features of the first word "happy" and the text encoding features of other words "delighted" is determined to be 0.9, and the second similarity between the text encoding features of the first word "happy" and the text encoding features of other words "heart is blossoming" is 0.7.

[0109] In step 2032A, the second similarities are averaged to obtain a similarity mean of the first word.

[0110] Continuing with the above example, the second similarity between the text encoding features of the first word "happy" and the text encoding features of other words "delighted" is 0.9, and the second similarity between the text encoding features of the first word "happy" and the text encoding features of other words "heart-blooming" is 0.7. The second similarities are averaged, and the mean similarity of the first word "happy" is 0.8, which is the average of the second similarity of 0.9 and the second similarity of 0.7.

[0111] In step 2033A, the first word corresponding to the maximum similarity mean is determined as the base word of the candidate synonym group.

[0112] Continuing with the above example, the mean similarity of "happy" is 0.8. When the second similarity between the text encoding feature of "delighted" and the text encoding feature of "heart is overjoyed" is 0.5, according to steps 2031A to 2032A, the mean similarity of the first word "delighted" is determined to be 0.7, and the mean similarity of the first word "heart is overjoyed" is determined to be 0.6. The first word "happy" corresponding to the maximum similarity mean of 0.8 is determined as the benchmark word of the candidate synonym group.

[0113] In an embodiment of the present application, by calculating the second similarity between the first word and other words, the semantic relationship between each word within the synonym group can be more accurately evaluated, thereby improving the accuracy of determining the benchmark word. At the same time, by averaging the second similarities between each first word and other words, the similarity of each first word can be quickly obtained, thereby improving the efficiency of determining the benchmark word.

[0114] In some embodiments, see Figure 3D , Figure 3D is a fourth flow chart of the data processing method provided in the embodiment of the present application, Figure 3B The step 203 of "selecting a reference word from the first word included in the candidate synonym group" shown in the figure can be implemented by the following steps 2031B to 2033B, which are described in detail below.

[0115] In step 2031B, text encoding processing is performed on each first word in the candidate synonym group to obtain a text encoding feature of each first word.

[0116] As an example, the specific steps of text encoding processing refer to the method of performing text encoding processing on the word to be processed mentioned above, which will not be described in detail here.

[0117] In step 2032B, the text encoding features of the plurality of first words are averaged to obtain the phrase encoding features of the candidate synonymous phrase.

[0118] As an example, the text encoding feature of the first word includes feature values ​​of multiple dimensions, and the average value of the feature value of each dimension of the text encoding feature of the first word is calculated to obtain a phrase encoding feature vector.

[0119] For example, for the candidate synonym group {"happy", "delighted", "heart blossoming"}, text encoding processing is performed on each first word in the candidate synonym group, and the text encoding features of "happy" are obtained as [feature 1, feature 2...feature n], the text encoding features of "delighted" are [feature 1', feature 2'...feature n'], and the text encoding features of "heart blossoming" are [feature 1", feature 2"...feature n"], and for each feature dimension, the average of the feature values ​​of the three words is calculated, and the obtained phrase encoding feature vector is [average feature 1, average feature 2...average feature n], where average feature 1 is the average of feature 1, feature 1' and feature 1", average feature 2 is the average of feature 2, feature 2' and feature 2", and average feature n is the average of feature n, feature n' and feature n".

[0120] In step 2033B, the third similarity between the phrase coding feature and the text coding feature of each first word is determined, and the first word corresponding to the maximum third similarity is determined as the benchmark word of the candidate synonym group.

[0121] As an example, determine the third similarity between the phrase coding feature and the text coding feature of each first word, determine the maximum third similarity among all the determined third similarities, and determine the first word corresponding to the maximum third similarity as the benchmark word of the candidate synonym group, wherein the specific steps for determining the third similarity refer to the method for determining the similarity between the first coding feature and the second coding feature above, which will not be repeated here.

[0122] In the embodiment of the present application, by selecting the first word corresponding to the maximum third similarity as the reference word, the semantic consistency between the selected reference word and the synonym group can be ensured, providing a reliable basis for subsequent text analysis and processing.

[0123] In step 103, when any first matching degree is greater than a first matching degree threshold, the word to be processed is written into a first synonym group, and an evaluation index of the first synonym group is determined.

[0124] The first synonym group is a candidate synonym group corresponding to any first matching degree, and the evaluation index is used to represent the probability of the first synonym group being added to the formal dictionary.

[0125] As an example, when any first matching degree is greater than a first matching degree threshold, the candidate synonym group corresponding to any first matching degree is determined as the first synonym group; when the first synonym group includes one synonym group, the word to be processed is written into the first synonym group, and an evaluation index of the first synonym group is determined; when the first synonym group includes at least two synonym groups, the word to be processed is written into the first synonym group corresponding to the highest first matching degree, and an evaluation index of the first synonym group corresponding to the highest first matching degree is determined, wherein the evaluation index may include at least one of the number of hits, the text association coefficient, and the word similarity.

[0126] In some embodiments, see Figure 3E , Figure 3E is a fifth flow chart of the data processing method provided in the embodiment of the present application, Figure 3A The "determining the evaluation index of the first synonym group" in step 103 shown in the figure can be implemented by the following steps 1031 to 1033, which are described in detail below.

[0127] In step 1031, when the evaluation index includes the number of hits, the number of hits of the first synonym group is incremented.

[0128] As an example, the number of hits is the number of times the first synonym group is successfully matched. When any first matching degree is greater than the first matching degree threshold, it indicates that the number of times the first synonym group is successfully matched increases. When the evaluation indicator includes the number of hits, the number of hits of the first synonym group is value-added, wherein the value-added processing includes adding one to the number of hits of the first synonym group.

[0129] In step 1032, when the evaluation index includes a text association coefficient, a prompt text is constructed based on the first synonym group and the text information, and a correlation coefficient between the first synonym group and the text information is predicted based on the prompt text to obtain the text association coefficient.

[0130] The words to be processed are extracted from text information.

[0131] As an example, the text association coefficient refers to the degree of association between the first synonym group and the text information. When the evaluation index includes the text association coefficient, a prompt text is constructed based on the first synonym group and the text information, and the large language model is called based on the prompt text to predict the association coefficient between the first synonym group and the text information to obtain the text association coefficient output by the large language model, wherein the prompt text is used to guide the large language model to predict the text association coefficient between the first synonym group and the text information. The embodiment of the present application does not limit the model structure of the large language model. For example, the large language model can be a mixture model (Mixture Model) or a generative pre-trained transformer model (GPT, Generative Pre-trained Transformer).

[0132] In the embodiment of the present application, by constructing a prompt text, more context information is provided to the large language model, thereby helping the model to more accurately predict the degree of association between the first synonym group and the text information.

[0133] In step 1033, when the evaluation index includes word similarity, at least one target word included in the target word configuration table is obtained, a first similarity between the first word included in the first synonym group and each target word is determined, and based on the first similarity, the word similarity is determined.

[0134] As an example, the target word configuration table is a configurable word list, which includes at least one target word. The word similarity refers to the semantic similarity between the first word and the target word. For example, the target word configuration table can be {"pleasure", "healthy life"}.

[0135] When the evaluation index includes word similarity, the first similarity between the text encoding feature of each first word and the text encoding feature of each target word is determined, and the word similarity is determined based on the first similarity, wherein determining the word similarity based on the first similarity includes determining the word similarity based on the maximum first similarity, and determining the word similarity based on the average of the first similarities.

[0136] In the embodiment of the present application, a configurable target word configuration table is used to achieve dynamic adjustment of target words based on different task requirements or business scenarios, thereby increasing the flexibility and customization capability of the evaluation indicators.

[0137] In step 104 , based on the evaluation index, the first synonym group is moved from the candidate dictionary to the formal dictionary.

[0138] As an example, when the evaluation index of the first synonym group in the candidate dictionary is greater than the index threshold, based on the number of the first words in the candidate synonym group, a corresponding benchmark word determination strategy is selected, and based on the corresponding benchmark word determination strategy, the benchmark word that best represents the entire candidate synonym group is reselected in the candidate synonym group, and then the first synonym group is moved from the candidate dictionary to the formal dictionary, wherein the specific steps of "reselecting the benchmark word that best represents the entire candidate synonym group in the candidate synonym group based on the corresponding benchmark word determination strategy" refer to steps 201 to 203 above, which will not be repeated here.

[0139] In some embodiments, Figure 3A Step 104 shown in the figure can be implemented by the following steps: when the evaluation index is greater than the index threshold, the first synonym group is determined as a new formal synonym group, the new formal synonym group is written into the formal dictionary, and the first synonym group in the candidate dictionary is deleted.

[0140] As an example, the evaluation indicators may include the number of hits, the text association coefficient and the word similarity. When the number of hits is greater than the hit number threshold, or the text association coefficient is greater than the text association coefficient threshold, or the word similarity is greater than the word similarity threshold, the first synonym group is determined as a new formal synonym group, and the new formal synonym group is written into the formal dictionary, and the first synonym group in the candidate dictionary is deleted.

[0141] The embodiment of the present application uses a dynamically updated synonym group management mechanism to dynamically adjust synonym groups in combination with evaluation indicators, thereby ensuring that the official dictionary can always reflect the latest developments and trends in the current field, thereby improving the accuracy and real-time performance of the official dictionary.

[0142] In some embodiments, see Figure 3F , Figure 3F is a sixth flow chart of the data processing method provided in the embodiment of the present application, Figure 3A After step 101 shown, the following steps 105 to 107 may be performed, which are described in detail below.

[0143] In step 105, a second matching degree between the word to be processed and each formal synonym group is determined.

[0144] As an example, the specific steps of determining the second matching degree refer to the method of determining the first matching degree in step 102 above, which will not be described in detail here.

[0145] In step 106, when any second matching degree is greater than a second matching degree threshold, the word to be processed is written into the second synonym group.

[0146] The second synonym group is a formal synonym group corresponding to any second matching degree.

[0147] As an example, when any second matching degree is greater than the second matching degree threshold, the word to be processed is written into the second synonym group, and an evaluation index of the second synonym group is determined. For specific steps, refer to step 103 above, which will not be described in detail here.

[0148] In step 107, when the second matching degree between the word to be processed and each formal synonym group is less than or equal to the second matching degree threshold, the word to be processed is matched with each candidate synonym group.

[0149] As an example, when the second matching degree between the word to be processed and each formal synonym group is less than or equal to the second matching degree threshold, it indicates that the word to be processed fails to match the formal dictionary, and step 102 is continued to match the word to be processed with the formal dictionary.

[0150] In some embodiments, see Figure 3G , Figure 3G is a seventh flow chart of the data processing method provided in the embodiment of the present application, Figure 3A After step 104 shown, the following steps 108 to 109 may be performed, which are described in detail below.

[0151] In step 108, when the first matching degree between the word to be processed and each candidate synonym group is less than or equal to the first matching degree threshold, a new candidate synonym group is created based on the word to be processed.

[0152] As an example, when the first matching degree between the word to be processed and each candidate synonym group is less than or equal to the first matching degree threshold, it indicates that the word to be processed fails to match the candidate dictionary, and a new candidate synonym group is created based on the word to be processed, wherein the new candidate synonym group includes the word to be processed, and the base word of the new candidate synonym group is the word to be processed.

[0153] In step 109, the new candidate synonym group is written into the candidate dictionary.

[0154] In an embodiment of the present application, when the first degree of match between the word to be processed and each candidate synonym group is less than or equal to the first degree of match threshold, a new candidate synonym group is automatically created, which reduces the burden of manual annotation and maintenance, and not only improves the coverage and flexibility of the candidate dictionary, but also improves the generation efficiency of synonym groups in the candidate dictionary.

[0155] Below, an exemplary application of the embodiment of the present application in an actual application scenario of generating a mobile application domain dictionary (APP domain dictionary) will be described.

[0156] In the field of natural language processing, a synonym domain dictionary is a word resource that focuses on a specific field. It not only contains standard terms in the field, but also includes other words with the same or very similar meanings as these terms, forming a synonym set, that is, a synonym group. In related technologies, synonym groups are constructed based on pre-set rules and static word knowledge bases. For example, researchers usually use ready-made synonym dictionaries or word form restoration technology to derive synonym groups. However, related technologies are difficult to generate accurate synonyms for domain-specific professional terms, or lack effective management and continuous update mechanisms for synonym groups in word domain dictionaries. They are unable to cope with large-scale and continuously evolving data sets, resulting in poor accuracy and real-time performance of the dictionary.

[0157] In order to solve the above problems, the embodiment of the present application proposes a data processing method with high accuracy and real-time performance, which is conducive to improving the information classification and labeling effects in mobile applications (APPs), and has important applications in multiple fields such as comments and advertising. The key technical points of the data processing method provided in the embodiment of the present application are introduced below.

[0158] 1) Creation and management of domain dictionaries: Two dictionaries are designed in the data processing method of the embodiment of the present application, namely, a candidate dictionary (candidate_index) and a formal dictionary (formal_index, equivalent to the formal dictionary above). The candidate dictionary is used to store synonym groups to be evaluated, and the formal dictionary is used to store confirmed synonym groups. This structure helps to realize dynamic updating and management of synonym groups, and improve the accuracy and real-time performance of the dictionary.

[0159] 2) Dynamic update: Determine whether the synonym group has reached a certain maturity based on the evaluation indicators, and then realize the migration of synonym groups between candidate dictionaries and formal dictionaries, as well as the update within the synonym group.

[0160] 3) Embedded computing method to determine synonym similarity: The input label (equivalent to the word to be processed above) is converted into a vector representation through a text embedding model (Embedding model) to capture its semantic information for subsequent similarity calculation. The similarity calculation uses methods such as cosine similarity to calculate the similarity between the input label and the synonym group in the dictionary. The preset threshold is used to determine whether the label is similar to the existing synonym group benchmark word, thereby deciding whether to add it to the synonym group.

[0161] 4) Base word selection: In a synonym group, a base word is selected as the base word of the synonym group based on criteria such as word length and mean cosine similarity. This helps to improve the representativeness of the base word in the synonym group and facilitates subsequent relevance detection. By selecting a base word based on criteria such as word length and mean cosine similarity, the problem of overfitting can be prevented during dictionary expansion.

[0162] The data processing method of the embodiment of the present application combines embedded computing and nested synonym storage to achieve automatic and real-time synonym generation and management; dynamically updates synonym groups through evaluation indicators to improve the accuracy and real-time performance of the dictionary; and selects benchmark words using standards such as the mean cosine similarity value, so that the benchmark words of the synonym group have better representativeness.

[0163] Next, the data processing method of the embodiment of the present application will be described from the product side by taking the generation of a domain dictionary of an instant messaging application as an example.

[0164] 1) Comment relevance judgment: see Figure 4 , Figure 4 It is the first principle diagram of the data processing method provided by the embodiment of the present application, and the real-time stream 402 of application comments (real-time stream of instant messaging APP comments, equivalent to the text information above) is subjected to natural language processing, and the subject corresponding to the user comments in the real-time stream of comments is extracted, and the subject of the user comments is matched with the application field dictionary 401, so as to judge the relevance between the user comments and the current APP, and add relevance tags to the corresponding user comments according to the relevance judgment results "whether relevant, hit synonym groups, etc.", and store the user comments with relevance tags in the comment database 403, wherein the application field dictionary 401 can be [{"communication tool", "instant messaging"}, {"social media", "social platform"}, {"post", "post posting", "post sharing"}], and the relevance tag can include the synonym group identity (ID) hit by the application field dictionary 401, whether relevant, relevance score, etc. Comment relevance judgment This helps to screen out comments related to the APP content and provide users with more accurate and targeted comment information.

[0165] 2) Hot Comment Topic Mining: See Figure 5 , Figure 5This is the second principle diagram of the data processing method provided in the embodiment of the present application. After the corresponding user comments are added with relevance labels (identity identification of the synonymous phrases hit, whether they are relevant, scores, etc.) through the processing of the relevance judgment module 501, the number of times the subject hits each phrase in the domain dictionary of the user comments in the comment database 502 is calculated, and the topics are sorted by subject hits according to the number of subject hits to obtain a hot topic list 503. The number of subject hits is used as the basis for evaluating hot topics, thereby helping application developers to timely understand the focus of user attention and provide data support for product optimization and operational decisions.

[0166] 3) Topic-based review retrieval: see Figure 6 , Figure 6 This is the third principle diagram of the data processing method provided by the embodiment of the present application. After the corresponding user comments are added with relevance tags (identity of the synonym group hit, whether it is relevant, score, etc.) through the processing of the relevance judgment module 601, the user comments with the same theme are stored in the comment database 602 based on the identity list 604 of the theme, and the function of reverse query of user comments by theme is realized. Specifically, the comments containing the identity of a certain theme are searched from the comment database 602 to obtain the comment list 603 of the corresponding theme, wherein the user comments with the same theme refer to the user comments with the same synonym group hit by the theme. For example, when the user clicks the "post" keyword (theme), the system can return user comments related to the post to improve the user experience.

[0167] 4) User content driven dictionary update: The topics extracted from user comments are added to the corresponding synonym groups in the candidate dictionary according to the data processing method of the embodiment of the present application. When the synonym groups in the candidate dictionary meet certain conditions, the synonym groups are migrated to the formal dictionary. In this way, potential synonym groups can be extracted from user-generated content, making the domain dictionary closer to actual application scenarios and improving the accuracy and real-time performance of the domain dictionary.

[0168] See also Figure 7 , Figure 7 This is the fourth principle diagram of the data processing method provided in the embodiment of the present application. Below, the process overview of the data processing method in the embodiment of the present application will be explained from the technical side.

[0169] First, initialize the dictionary, create candidate dictionaries and formal dictionaries; at the same time, prepare the original corpus (equivalent to the text information above), and use natural language processing methods to perform fine-grained sentence segmentation on the original corpus, for example, short text segmentation, and perform topic extraction on each sentence of the original corpus, and use the extracted labels 1, 2, and 3 as input labels (equivalent to the words to be processed above). Natural language processing includes: processing through a natural language model (NLP model), processing through a bidirectional pre-trained language model (BERT model), and processing through a pre-trained language model (GPT model). The original corpus can be application description information, instructions for use, high-quality community information, etc. in the application field. The candidate dictionary is used to store synonym groups to be evaluated, and the formal dictionary is used to store confirmed synonym groups.

[0170] Then, embedded calculations are performed on the input label 1, label 2, and label 3 to obtain corresponding vector representations, where the vector representation is usually a matrix of 1024*768 dimensions, and the specific dimension depends on the dimensional size of the embedded calculation model.

[0171] Finally, after calculating the vector representation of the input label, the vector representation of the input label is matched with the vector representation of the benchmark words of each synonym group in the formal dictionary to obtain the similarity between the vector representation of the input label and the vector representation of the benchmark words of each synonym group in the formal dictionary. If the similarity between the vector representation of the input label and the vector representation of the benchmark word of a synonym group in the formal dictionary meets the standard (the similarity reaches the threshold, score>threshotd), the formal dictionary match is successful, and the label is added to the formal dictionary, that is, the hit label (label 1 and label 2) is directly selected into the synonym group; if the synonym group in the formal dictionary is not selected, the missed word label 3 is matched with the vector representation of the benchmark words of each synonym group in the candidate dictionary to obtain the vector representation of label 3 and the benchmark words of each synonym group in the candidate dictionary. The similarity between the vector representations of the benchmark words of the group. If the similarity between the vector representation of label 3 and the vector representation of the benchmark words of the synonym group in the candidate dictionary reaches the threshold, the candidate dictionary is matched successfully, and label 3 is directly selected into the synonym group. If the evaluation index of the synonym group meets the standard (hitCount>threshold?), for example, the number of hits of the synonym group reaches the hit number threshold (the number of hits meets the standard), the current synonym group is migrated to the formal dictionary. At the same time, based on the root selection algorithm, the benchmark word that best represents the entire synonym group is reselected in the current synonym group. If the evaluation index of the synonym group does not meet the standard at this time, the evaluation index of the synonym group is updated. For example, if the number of hits does not meet the standard, only the number of hits recorded in the candidate dictionary is increased by one. Among them, a regular review mechanism can be designed for synonym groups that enter the formal dictionary from the candidate dictionary.

[0172] The following is a detailed description of the technical details of the data processing method in the embodiment of the present application.

[0173] 1) Vector Representation

[0174] The vector representation of the input label (the vector representation of the benchmark word) can be realized by processing the input label (benchmark word) through a traditional word vector model (Word2Vec), a global word vector model (GloVe model) and other word vector models, or it can be realized through a larger embedding model such as a Hunyuan Embedding model and a large embedding model (GBE Large Embedding), which is not limited here.

[0175] 2) Dictionary mode combining candidate dictionary and formal dictionary

[0176] See also Figure 8 , Figure 8 This is the fifth principle diagram of the data processing method provided in the embodiment of the present application. The dictionary creation part includes two dictionaries: a candidate dictionary and a formal dictionary. The dictionary mode combining the candidate dictionary and the formal dictionary is designed to realize the dynamic update and management of synonym groups and improve the accuracy and real-time performance of the dictionary.

[0177] Candidate dictionary: used to store synonym groups to be evaluated, see Figure 8 When a new tag (missing tag) tag 3 is input, firstly, the similarity between tag 3 and the vector representation of the benchmark word of the synonym group in the candidate dictionary is calculated to obtain the similarity between the vector representation of tag 3 and the vector representation of the benchmark word of each synonym group in the candidate dictionary. If the similarity exceeds the preset threshold, it means that tag 3 may have similar semantics to a synonym group, and tag 3 can be added to the synonym group. At the same time, according to the evaluation index of the synonym group (for example, the number of hits), it is judged whether the synonym group has reached a certain maturity. When the synonym group reaches maturity, the synonym group is migrated from the candidate dictionary to the formal dictionary. For example, if phrase 1 hits more than 5 times, it means that phrase 1 is often mentioned in the application, and phrase 1 is directly written into the formal dictionary.

[0178] Formal dictionary: used to store confirmed synonym groups, see Figure 8When label 1, label 2 and label 3 are input, the similarity between them and the vector representation of the benchmark words of the synonym group in the formal dictionary is calculated to obtain the similarity between the vector representation of the input label and the vector representation of the benchmark words of each synonym group in the formal dictionary, and judge whether the similarity meets the standard. If the similarity exceeds the preset threshold, it means that the label has similar semantics to a synonym group, and it can be added to the synonym group, and the evaluation index of the synonym group is updated.

[0179] 3) Adding logic to phrase entities

[0180] When a new tag is input, the similarity between the tag and the synonym groups in the formal dictionary and the candidate dictionary is calculated. If the similarity between the tag and a synonym group in the formal dictionary exceeds a threshold, the tag is added to the synonym group and the evaluation index of the synonym group is updated; if the similarity with a synonym group in the candidate dictionary exceeds a threshold, the tag is added to the synonym group, and based on the evaluation index of the synonym group (for example, the number of hits), it is determined whether the synonym group has reached a certain maturity. If so, the synonym group is migrated from the candidate dictionary to the formal dictionary.

[0181] If the similarity between a tag and synonym groups in both the candidate dictionary and the formal dictionary does not exceed the threshold, the tag is added to the candidate dictionary as a new synonym group.

[0182] 4) Similarity calculation method

[0183] In natural language processing, similarity calculation of embedded vectors is a key step, which can be used to measure the similarity between two words in the semantic space. Common similarity calculation methods include Euclidean distance, Mahalanobis distance, Pearson correlation coefficient, Jaccard similarity coefficient, cosine similarity, etc. The embodiment of the present application adopts the cosine similarity calculation method to calculate the similarity between two words in the semantic space.

[0184] Cosine similarity is a similarity calculation method based on vector space. It measures the similarity between two vectors by calculating the cosine value of the angle between the two vectors. Specifically, cosine similarity can be calculated as the ratio between the dot product of the two vectors and the module-length product of the two vectors.

[0185] The reasons why the embodiment of the present application selects cosine similarity as the similarity calculation method of embedded vectors are as follows:

[0186] First, cosine similarity is insensitive to the length of the vector and only focuses on the direction of the vector. This means that even if the lengths of two vectors (i.e., the frequencies of words) are different, as long as their directions are the same (i.e., semantic similarity), the cosine similarity between the two vectors will be very high. This is very important for processing text data, because in text, the frequency of words is often affected by many factors, and semantic similarity can better reflect the actual relationship between words.

[0187] Secondly, cosine similarity is computationally efficient and suitable for processing large-scale data. In high-dimensional space, calculating the cosine of the angle between two vectors is usually faster than calculating their Euclidean distance or Mahalanobis distance.

[0188] Finally, processing cosine similarity is widely used in natural language and has been shown to achieve good results in many tasks, such as document classification, information retrieval, word sense disambiguation, etc.

[0189] 5) Benchmark word selection algorithm (root selection algorithm).

[0190] The embodiment of the present application provides an algorithm for selecting a benchmark word in a synonym group. When the evaluation index of the synonym group in the candidate dictionary reaches the standard, the synonym group is moved from the candidate dictionary to the supporting formal dictionary. The benchmark word that best represents the entire synonym group is reselected from the synonym group through a root selection algorithm. Fig. 9 , Fig. 9 This is the sixth principle diagram of the data processing method provided in the embodiment of the present application. When the candidate synonym group is determined to be able to be written into the formal dictionary, the specific implementation is as follows:

[0191] In step 901, it is determined whether the candidate synonym group can be written into the formal dictionary. When the candidate synonym group can be written into the formal dictionary, step 902 is executed to determine the length of the candidate synonym group. If the synonym group is empty, an exception prompt is thrown. If the candidate synonym group contains only one word (the number of words in the candidate synonym group = 1), step 903 is executed to use the only word as the benchmark word; if the candidate synonym group contains two words (the number of words in the candidate synonym group = 2), it is determined whether the lengths of the two words are consistent. If the lengths of the two words are inconsistent, step 904 is executed to select the shorter word as the benchmark word. If the lengths of the two words are consistent, step 905 is executed to select the first word as the benchmark word or randomly select the benchmark word; if the candidate synonym group contains three or more words (the number of words in the candidate synonym group ≥ 3), step 906 is executed to calculate the mean cosine similarity of each word in the candidate synonym group, and select the word with the cosine similarity closest to 1 as the benchmark word. Through the above algorithm, a representative benchmark word can be selected from the synonym group to facilitate subsequent relevance detection and dictionary generation.

[0192] The technical solution provided in the embodiment of the present application has a high degree of automation, accuracy and real-time performance, and can provide strong support for application scenarios such as comment systems, classification and labeling in the APP field, thereby producing the following significant beneficial effects:

[0193] Improve the degree of automation of dictionary generation: The APP domain dictionary generation method based on embedded computing and search engine (or data storage and retrieval system) in the embodiment of the present application can automatically generate a domain dictionary for the APP, reduce the burden of manual annotation and maintenance, and improve the efficiency of dictionary generation.

[0194] Improve the accuracy and real-time performance of the dictionary: The embodiment of the present application adopts a dynamically updated synonym group management mechanism, and dynamically adjusts the synonym groups in combination with evaluation indicators (such as comment rate), ensuring that the dictionary can always reflect the latest developments and trends in the APP field, thereby improving the accuracy and real-time performance of the dictionary.

[0195] Improve the accuracy of correlation detection: The embodiment of the present application can accurately measure the semantic similarity between words by calculating the cosine similarity of embedded vectors, thereby providing an accurate correlation detection basis for application scenarios such as the APP review system.

[0196] Improve user experience: The APP domain dictionary generated by the embodiment of the present application can help users find APP-related information more accurately and quickly, and improve the user experience quality in application scenarios such as comment systems.

[0197] Promote the development of APP classification and labeling: The embodiment of the present application provides an efficient and accurate method for generating a dictionary in the APP field, which helps to promote the development of APP classification and labeling technology and bring more convenience to APP developers and users.

[0198] The following further describes an exemplary structure of the data processing device 555 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the data processing device 555 of the memory 550 may include:

[0199] The data acquisition module 5551 is used to acquire the word to be processed, the formal dictionary and the candidate dictionary, wherein the formal dictionary includes at least one formal synonym group, the candidate dictionary includes at least one candidate synonym group, and the formal synonym group and the candidate synonym group are both sets of synonyms.

[0200] The matching degree determination module 5552 is used to determine the first matching degree between the word to be processed and each of the candidate synonym groups.

[0201] The data writing module 5553 is used to write the word to be processed into the first synonym group when any of the first matching degrees is greater than a first matching degree threshold, and determine an evaluation index for the first synonym group, wherein the first synonym group is a candidate synonym group corresponding to any of the first matching degrees, and the evaluation index is used to represent the probability of the first synonym group being added to the formal dictionary.

[0202] The data transmission module 5554 is used to move the first synonym group from the candidate dictionary to the formal dictionary based on the evaluation index.

[0203] In some embodiments, the matching degree determination module 5552 is further used to determine the second matching degree between the word to be processed and each of the formal synonym groups; the data writing module 5553 is further used to write the word to be processed into the second synonym group when any of the second matching degrees is greater than a second matching degree threshold, wherein the second synonym group is a formal synonym group corresponding to any of the second matching degrees; the matching degree determination module 5552 is further used to match the word to be processed with each of the candidate synonym groups when the second matching degrees between the word to be processed and each of the formal synonym groups are less than or equal to the second matching degree threshold.

[0204] In some embodiments, the data writing module 5553 is also used to create a new candidate synonym group based on the word to be processed when the first matching degree between the word to be processed and each of the candidate synonym groups is less than or equal to the first matching degree threshold; and write the new candidate synonym group into the candidate dictionary.

[0205] In some embodiments, the data writing module 5553 is also used to, when the evaluation index includes the number of hits, perform value-added processing on the number of hits of the first synonym group, wherein the number of hits is the number of times the first synonym group is successfully matched; when the evaluation index includes the text association coefficient, construct a prompt text based on the first synonym group and the text information, and predict the association coefficient between the first synonym group and the text information based on the prompt text to obtain the text association coefficient, wherein the word to be processed is extracted from the text information; when the evaluation index includes word similarity, obtain at least one target word included in the target word configuration table, determine the first similarity between the first word included in the first synonym group and each of the target words, and determine the word similarity based on the first similarity.

[0206] In some embodiments, the matching degree determination module 5552 is further used to perform text encoding on the word to be processed to obtain a first encoding feature of the word to be processed; perform text encoding on the benchmark word of each candidate synonym group to obtain a second encoding feature of each benchmark word; and use the similarity between the first encoding feature and the second encoding feature as the first matching degree.

[0207] In some embodiments, the matching degree determination module 5552 is further used to perform the following processing for each candidate synonym group: determine the number of first words included in the candidate synonym group; and determine the benchmark word of the candidate synonym group based on the number of the first words.

[0208] In some embodiments, the matching degree determination module 5552 is further used to, when the number of the first words is equal to 1, use the first word as the base word of the candidate synonym group; when the number of the first words is equal to 2, determine the first word with the least number of characters as the base word of the candidate synonym group; when the number of the first words is greater than 2, select the base word from the first words included in the candidate synonym group.

[0209] In some embodiments, the matching degree determination module 5552 is further used to perform the following processing on any first word in the candidate synonym group: determine the second similarity between the first word and other words, wherein the other words are words in the candidate synonym group other than the first word; average the second similarities to obtain the mean similarity of the first words; and determine the first word corresponding to the maximum similarity mean as the benchmark word of the candidate synonym group.

[0210] In some embodiments, the matching degree determination module 5552 further performs text encoding processing on each of the first words in the candidate synonym group to obtain text encoding features of each of the first words; averages the text encoding features of multiple first words to obtain the phrase encoding features of the candidate synonym group; determines the third similarity between the phrase encoding features and the text encoding features of each of the first words, and determines the first word corresponding to the maximum third similarity as the benchmark word of the candidate synonym group.

[0211] In some embodiments, the data transmission module 5554 is further used to, when the evaluation index is greater than an index threshold, determine the first synonym group as a new formal synonym group; write the new formal synonym group into the formal dictionary, and delete the first synonym group in the candidate dictionary.

[0212] In some embodiments, the data acquisition module 5551 is also used to acquire text information, and perform word segmentation processing on the text information to obtain at least one word segmentation; perform stop word removal processing on the at least one word segmentation to obtain the word segmentation after removal; based on the frequency of occurrence of the word segmentation after removal in the text information, determine the word to be processed from the word segmentation after removal.

[0213] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data processing method described above in the embodiment of the present application.

[0214] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the data processing method provided by the present application embodiment, for example, Figures 3A to 3G The data processing method is shown.

[0215] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0216] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0217] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0218] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0219] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain a word to be processed, a formal dictionary, and a candidate dictionary, wherein the formal dictionary includes at least one formal synonym group, the candidate dictionary includes at least one candidate synonym group, and both the formal synonym group and the candidate synonym group are sets of synonyms; Determining a first matching degree between the word to be processed and each of the candidate synonym groups; When any of the first matching degrees is greater than a first matching degree threshold, the word to be processed is written into a first synonym group, and an evaluation index of the first synonym group is determined, wherein the first synonym group is a candidate synonym group corresponding to any of the first matching degrees, and the evaluation index is used to represent the probability of the first synonym group being added to the formal dictionary; Based on the evaluation index, the first synonym group is moved from the candidate dictionary to the official dictionary.

2. The method according to claim 1, characterized in that The method further comprises: Determining a second matching degree between the word to be processed and each of the formal synonym groups; When any of the second matching degrees is greater than a second matching degree threshold, the word to be processed is written into a second synonym group, wherein the second synonym group is a formal synonym group corresponding to any of the second matching degrees; When the second matching degree between the word to be processed and each of the formal synonym groups is less than or equal to the second matching degree threshold, the word to be processed is matched with each of the candidate synonym groups.

3. The method according to claim 1, characterized in that The method further comprises: When the first matching degree between the word to be processed and each of the candidate synonym groups is less than or equal to the first matching degree threshold, creating a new candidate synonym group based on the word to be processed; The new candidate synonym group is written into the candidate dictionary.

4. The method according to any one of claims 1 to 3, characterized in that: The step of determining the evaluation index of the first synonym group includes: When the evaluation index includes the number of hits, the number of hits of the first synonym group is incremented, wherein the number of hits is the number of times the first synonym group is successfully matched; When the evaluation index includes a text association coefficient, a prompt text is constructed based on the first synonym group and the text information, and a correlation coefficient between the first synonym group and the text information is predicted based on the prompt text to obtain the text association coefficient, wherein the word to be processed is extracted from the text information; When the evaluation index includes word similarity, at least one target word included in the target word configuration table is obtained, a first similarity between a first word included in the first synonym group and each of the target words is determined, and the word similarity is determined based on the first similarity.

5. The method according to any one of claims 1 to 3, characterized in that: The determining of the first matching degree between the word to be processed and each of the candidate synonym groups includes: Performing text encoding on the word to be processed to obtain a first encoding feature of the word to be processed; Performing text encoding on the benchmark word of each candidate synonym group to obtain a second encoding feature of each benchmark word; The similarity between the first encoding feature and the second encoding feature is used as the first matching degree.

6. The method according to claim 5, characterized in that Before performing text encoding on the reference word of each candidate synonym group to obtain a second encoding feature of each reference word, the method further includes: The following processing is performed for each candidate synonym group: determining the number of first words included in the candidate synonym group; Based on the number of the first words, a base word of the candidate synonym group is determined.

7. The method according to claim 6, characterized in that The step of determining the benchmark word of the candidate synonym group based on the number of the first words includes: When the number of the first words is equal to 1, the first words are used as the base words of the candidate synonym group; When the number of the first words is equal to 2, determining the first word with the least number of characters as the base word of the candidate synonym group; When the number of the first words is greater than 2, the reference word is selected from the first words included in the candidate synonym group.

8. The method according to claim 7, characterized in that The step of selecting the reference word from the first word included in the candidate synonym group includes: The following processing is performed for any first word in the candidate synonym group: determining a second similarity between the first word and other words, wherein the other words are words in the candidate synonym group other than the first word; Performing an average processing on the second similarities to obtain a mean similarity of the first words; The first word corresponding to the maximum similarity mean is determined as the base word of the candidate synonym group.

9. The method according to claim 7, characterized in that: The step of selecting the reference word from the first word included in the candidate synonym group includes: Performing text encoding processing on each of the first words in the candidate synonym group to obtain a text encoding feature of each of the first words; Averaging the text encoding features of the plurality of first words to obtain the phrase encoding features of the candidate synonym phrase; A third similarity between the phrase encoding feature and the text encoding feature of each of the first words is determined, and the first word corresponding to the maximum third similarity is determined as the benchmark word of the candidate synonym group.

10. The method according to claim 1, characterized in that The step of moving the first synonym group from the candidate dictionary to the formal dictionary based on the evaluation index includes: When the evaluation index is greater than an index threshold, determining the first synonym group as a new formal synonym group; The new formal synonym group is written into the formal dictionary, and the first synonym group in the candidate dictionary is deleted.

11. The method according to claim 1, characterized in that: The step of obtaining the word to be processed includes: Acquire text information, and perform word segmentation processing on the text information to obtain at least one word segmentation; Performing stop word removal processing on the at least one segmented word to obtain the segmented word after the removal; Based on the occurrence frequency of the removed segmented words in the text information, the to-be-processed words are determined from the removed segmented words.

12. A data processing device, characterized in that: The device comprises: A data acquisition module, used to acquire a to-be-processed word, a formal dictionary, and a candidate dictionary, wherein the formal dictionary includes at least one formal synonym group, the candidate dictionary includes at least one candidate synonym group, and both the formal synonym group and the candidate synonym group are sets of synonyms; A matching degree determination module, used to determine a first matching degree between the word to be processed and each of the candidate synonym groups; a data writing module, configured to write the word to be processed into a first synonym group when any of the first matching degrees is greater than a first matching degree threshold, and determine an evaluation index for the first synonym group, wherein the first synonym group is a candidate synonym group corresponding to any of the first matching degrees, and the evaluation index is used to represent the probability of the first synonym group being added to the formal dictionary; A data transmission module is used to move the first synonym group from the candidate dictionary to the formal dictionary based on the evaluation index.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the data processing method according to any one of claims 1 to 11 when executing the computer executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

Images