Data processing method and related device

Abstract

The present application provides a data processing method and related devices, wherein the data processing method includes: obtaining an initial corpus data set corresponding to a first corpus category, and extracting semantic vectors for each corpus data in the initial corpus data set to obtain a semantic vector sequence for each corpus data, determining at least one semantic vector corresponding to the first corpus category according to the semantic vector sequence, and calculating the first occurrence number of each semantic vector under the first corpus category and the second occurrence number of each semantic vector under the second corpus category, and then determining the type of each semantic vector under the first corpus category according to the first occurrence number and the second occurrence number of each semantic vector, filtering the initial corpus data set according to the type to obtain the corpus data set. The use of the embodiments of the present application can improve the effectiveness of filtering the initial corpus data set.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and related devices. Background Art

[0002] With the continuous development and progress of the Internet and information technology, more and more services are being carried out through the Internet. As more and more data is processed on the Internet, various models can be used to assist in data processing to improve the efficiency and accuracy of data processing. With the development of diversified services, the processed data is becoming more and more diverse, and the types of data processing are also becoming more and more diverse. How to extract valuable data to train the model, thereby improving the processing effect of the model for data processing, and then improving the accuracy of the service, is a focus of increasing attention for service providers. Summary of the invention

[0003] The embodiments of the present application provide a data processing method and related devices to solve the problem of how to efficiently obtain more accurate and effective corpus data.

[0004] In a first aspect, an embodiment of the present application provides a data processing method, including: Acquire an initial corpus data set corresponding to the first corpus category, and extract semantic element vectors for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data; Determine at least one semantic element vector corresponding to the first corpus category according to the semantic element vector sequence, and calculate a first occurrence number of each semantic element vector in the first corpus category and a second occurrence number in the second corpus category; Determine the type of each sememe vector under the first corpus category according to the first number of occurrences and the second number of occurrences of each sememe vector; The initial corpus data set is filtered according to the type to obtain a corpus data set.

[0005] It can be seen that in the embodiment of the present application, during the data processing, for the initial corpus data set corresponding to the first corpus category, firstly, a semantic element vector is extracted for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data, and then at least one semantic element vector corresponding to the first corpus category is determined according to the semantic element vector sequence, and the first occurrence number of each semantic element vector under the first corpus category and the second occurrence number under the second corpus category are calculated, and then the type of each semantic element vector under the first corpus category is determined according to the first occurrence number and the second occurrence number of each semantic element vector; after obtaining at least one semantic element vector corresponding to the first corpus category, combined with each semantic element vector corresponding to the first corpus category, the semantic element vector is extracted. The first occurrence number of each semantic element vector under the first corpus category, and the second occurrence number of each semantic element vector corresponding to the first corpus category under the second corpus category, are used to determine the type of each semantic element vector under the first corpus category, and the accuracy of the type of each determined semantic element vector under the first corpus category is improved, and then the initial corpus data set is filtered according to the type of each semantic element vector to obtain the corpus data set. In this way, the corpus data is filtered starting from the semantic element vector. On the basis of realizing the filtering process of the initial corpus data set, it is ensured that the corpus data in the obtained corpus data set is more matched with the first corpus category, and the accuracy and effectiveness of the corpus data in the obtained corpus data set are improved.

[0006] In a second aspect, an embodiment of the present application provides a data processing device, including: A semantic element vector extraction module is used to obtain an initial corpus data set corresponding to the first corpus category, and extract semantic element vectors from each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data; a semantic element vector determination module, configured to determine at least one semantic element vector corresponding to the first corpus category according to the semantic element vector sequence, and calculate a first occurrence number of each semantic element vector in the first corpus category and a second occurrence number in the second corpus category; A type determination module, configured to determine the type of each sememe vector under the first corpus category according to the first number of occurrences and the second number of occurrences of each sememe vector; The filtering processing module is used to filter the initial corpus data set according to the type to obtain a corpus data set.

[0007] In a third aspect, an embodiment of the present application provides a data processing device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, cause the processor to execute the data processing method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing computer-executable instructions, which, when executed by a processor, implement the data processing method as described in the first aspect.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor. Figure 1 A schematic diagram of an implementation environment of a data processing method provided in an embodiment of the present application; Figure 2 A data processing method processing flow chart provided in an embodiment of the present application; Figure 3 A corpus tree diagram provided for an embodiment of the present application; Figure 4 A processing schematic diagram of a data processing method applied to a model training scenario provided in an embodiment of the present application; Figure 5 A schematic diagram of a data processing device provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0012] In practical applications, in order to process data effectively and efficiently, in some scenarios, the trained related models are used to process the corresponding data. However, in the process of model training, the performance of the models obtained by using different sample data for model training varies. In order to improve the effectiveness of the corpus data obtained for model training, the present embodiment provides a data processing method. For an initial corpus data set corresponding to each corpus category, the initial corpus data set is filtered according to the type of each semantic element vector corresponding to the corpus category to obtain a corpus data set. In this way, the corpus data in the initial corpus data set is filtered by filtering the semantic element dimension, and the initial corpus data set is effectively streamlined. In the specific implementation process, for an initial corpus data set corresponding to a first corpus category, at least one semantic element vector corresponding to the first corpus category is determined according to the semantic element vector sequence of each corpus data in the initial corpus data set, and the first occurrence number of each semantic element vector corresponding to the first corpus category under the first corpus category and the second occurrence number of each semantic element vector under the second corpus category are calculated. Further, the first occurrence number and the second occurrence number of each semantic element vector are used to determine the type of each semantic element vector under the first corpus category. In this way, the first occurrence number of each semantic element vector corresponding to the first corpus category under the first corpus category and the first occurrence number of each semantic element vector corresponding to the first corpus category under the second corpus category are combined. The type of each semantic element vector under the first corpus category is determined based on the second occurrence number under the first corpus category, and the accuracy of the type of each determined semantic element vector under the first corpus category is improved. Both the first occurrence number and the second occurrence number are calculated from the semantic element vector sequence of each corpus data corresponding to the first corpus category and the semantic element vector sequence of each corpus data corresponding to the second corpus category, respectively. Starting from the semantic element vector sequence of each corpus data in the initial corpus data set corresponding to each corpus category, the occurrence number of each semantic element vector corresponding to the corpus category is calculated, and the validity of the calculated first occurrence number and second occurrence number is improved. On the basis of improving the validity of the calculated first occurrence number and second occurrence number, the accuracy and validity of the type of each semantic element vector determined according to the first occurrence number and the second occurrence number under the first corpus category are further improved, thereby improving the validity of the filtering processing of the initial corpus data set according to the type, and on the basis of ensuring the matching degree between the corpus data in the obtained corpus data set and the first corpus category, the initial corpus data set is streamlined.

[0013] The data processing method provided in one or more embodiments of this specification can be applied to the implementation environment of corpus data processing, such as Figure 1 As shown, the implementation environment at least includes a server 101; The server 101 may be one or more servers, a server cluster consisting of several servers, or a cloud server of a cloud computing platform, and is used to filter the initial corpus data set corresponding to the first corpus category to obtain a corpus data set containing corpus data with a high degree of matching with the first corpus category; In addition, the implementation environment may also include a semantic vector model 102, a type recognition model 103 and an intention recognition model to be trained 104; The semantic element vector model 102 may be a pre-trained language model, such as a pre-trained language representation model (BERT, Bidirectional Encoder Representation from Transformers); the semantic element vector model may include a word recognition layer and a semantic element vector layer; the word recognition layer is used to recognize words in the corpus data, and the semantic element vector layer is used to call a semantic element vector acquisition interface to obtain semantic element vectors associated with words; The type identification model 103 is used to perform type identification according to the first occurrence number of each semantic element vector corresponding to the first corpus category in the first corpus category and the second occurrence number in the second corpus category, so as to obtain the type of each semantic element vector in the first corpus category; After obtaining corpus data sets of multiple corpus categories, the corpus data sets of each corpus category may be used to perform model training on the intention recognition model 104 to be trained; In this implementation environment, after the server 101 obtains the initial corpus data set corresponding to the first corpus category, it can first extract the semantic vector of each corpus data in the initial corpus data set through the semantic vector model 102 to obtain the semantic vector sequence of each corpus data, and then determine at least one semantic vector corresponding to the first corpus category based on the semantic vector sequence of each corpus data, and use the type recognition model 103 to identify the type of each semantic vector in the first corpus category according to the first occurrence number of each semantic vector in the first corpus category and the second occurrence number of each semantic vector in the second corpus category to obtain the type of each semantic vector in the first corpus category, and finally filter the initial corpus data set corresponding to the first corpus category according to the type to obtain the corpus data set, thereby improving the matching degree between the corpus data in the corpus data set of the first corpus category and the first corpus category.

[0014] Figure 2 A processing flow chart of a data processing method provided in this embodiment, refer to Figure 2 , a data processing method provided in this embodiment specifically includes steps S202 to S208; Step S202: acquiring an initial corpus data set corresponding to the first corpus category, and extracting semantic component vectors from each corpus data in the initial corpus data set to obtain a semantic component vector sequence of each corpus data.

[0015] The corpus data in this embodiment includes data that needs to be processed under different business requirements; for example, the corpus data can be user voice text obtained by performing voice recognition on the user voice after the service agency collects the user voice during a call with the user through a robot agent or a manual call; for another example, the corpus data can also be user text obtained after text interaction with the user; the above describes the case where the corpus data includes user voice text and / or user text in the process of interaction with the user. In addition, the corpus data can also be user data obtained by the service agency with the authorization of the user in the process of providing services, such as user data that characterizes the user identity and / or user status.

[0016] The corpus category includes dimensions that characterize the commonality of corpus data in the same initial corpus data set; for example, the corpus category may be an intent category that characterizes the intent of the corpus data in the initial corpus data set; for another example, the corpus category may be a preference category that characterizes the preference of the corpus data in the initial corpus data set; in addition, the corpus category may also be a state category and / or a demand category that characterizes the state and / or demand of the corpus data in the initial corpus data set; the corpus category in this embodiment may mark the corpus data according to service requirements, which is not limited in this embodiment. It should be noted that the corpus data and corpus category in this embodiment may be configured according to different businesses and different requirements, which is not limited in this embodiment.

[0017] At present, the research on natural language processing (NLP) and the exploration of model recognition are both stagnant at the word level. However, from the actual language situation, as the smallest unit that can be used independently, a word has a subordinate concept at the semantic level, namely, sememe (SEME). Sememe includes meaning units (content units) in language, also known as semesons, which are the constituent factors of words. In the comparison with different words, the same word can be expressed by different numbers of sememes according to usage requirements. For example, if the word "woman" is limited to three sememes, it can be expressed as [people] [female] [adult]. It can be seen that by analyzing sememes, the semantics of the text can be studied at a smaller granularity, so that the true meaning of each word in the text can be more accurately identified. Semantic features can distinguish multiple meanings of the same word, while sememes can decompose the same meaning of the same word, so that the semantics of the text can be studied at a smaller granularity. Among them, a word can correspond to one or more sememes. In addition, for polysemous words, different meanings may be presented in different sentences. Therefore, if there are polysemous words in a sentence, the semantic elements of the polysemous word can be determined based on the semantic elements of the words before and after the polysemous word. In some embodiments, a semantic element vector can be used to represent words. Semantics include the smallest semantic units that constitute semantics, and the semantic structure and relationship of words can be better understood through semantic element analysis. The semantic element vector in this embodiment includes a representation of semantic elements in machine language. In the case where a word can be decomposed into multiple semantic elements, the semantic element vector sequence of each corpus data is composed of the semantic element vector of at least one semantic element associated with each word contained in the corpus data.

[0018] In actual applications, there are multiple corpus categories, and each corpus category has a corresponding initial corpus data set. In this embodiment, the initial corpus data set corresponding to the first corpus category is filtered to obtain the corpus data set corresponding to the first corpus category. The first corpus category can be any corpus category. Correspondingly, the second corpus category includes one or more corpus categories other than the first corpus category among the multiple corpus categories.

[0019] In a specific implementation, for an initial corpus data set corresponding to the first corpus category, a semantic element vector is extracted for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data.

[0020] In order to improve the convenience of semantic vector extraction and to improve the accuracy of the semantic vector sequence of each corpus data, this embodiment can extract semantic vectors from each corpus data in the initial corpus data set based on the semantic vector library to obtain the semantic vector sequence of each corpus data; in an optional implementation manner provided by this embodiment, the process of extracting semantic vectors from each corpus data in the initial corpus data set to obtain the semantic vector sequence of each corpus data is implemented in the following manner: Performing word recognition on each corpus data to obtain at least one word contained in each corpus data; The semantic element vector associated with each word is read from the semantic element vector library, and a semantic element vector sequence of each corpus data is generated according to the semantic element vector associated with at least one word.

[0021] Specifically, in the process of extracting semantic component vectors for each corpus data, firstly, word recognition is performed on each corpus data to obtain at least one word contained in each corpus data, then the semantic component vector associated with each word is read from the semantic component vector library, and finally, according to the semantic component vector associated with at least one word, a semantic component vector sequence for each corpus data is generated.

[0022] Taking corpus data 1 as an example, word recognition is performed on corpus data 1, and corpus data 1 includes word 1, word 2 and word 3; the semantic element vectors associated with word 1, word 2 and word 3 are read in the semantic element vector library respectively, and the semantic element vector associated with word 1 includes {[semantic element vector 1], [semantic element vector 2]}; the semantic element vector associated with word 2 includes {[semantic element vector 1], [semantic element vector 3], [semantic element vector 4]}, and the semantic element vector associated with word 3 includes {[semantic element vector 1], [semantic element vector 2], [semantic element vector 4]}; Furthermore, the semantic vector sequence of corpus data 1 is generated based on the semantic vectors associated with word 1, word 2, and word 3: {{[semetic vector 1], [semetic vector 2]}, {[semetic vector 1], [semetic vector 3], [semetic vector 4]}, {[semetic vector 1], [semetic vector 2], [semetic vector 4]}}. It should be noted that the semantic vectors associated with different words may be completely or partially equal.

[0023] In addition, the semantic vector extraction of each corpus data in the initial corpus data set can also be implemented through a semantic vector model; the semantic vector model can be a pre-trained language model, such as a pre-trained language representation model; the semantic vector model can include a word recognition layer and a semantic vector layer, wherein the word recognition layer is used to recognize words in the corpus data, and the semantic vector layer is used to call the semantic vector acquisition interface to obtain the semantic vector associated with the word. Specifically, each corpus data in the initial corpus data set can be input into the semantic recognition model for semantic vector extraction to obtain a semantic vector sequence of each corpus data. Optionally, after obtaining the corpus data, the semantic vector model first performs word recognition on each corpus data through the word recognition layer to obtain at least one word contained in each corpus data, and then calls the semantic vector acquisition interface through the semantic vector layer to obtain the semantic vector associated with each word in the semantic vector library, and finally generates the semantic vector data of the corpus data according to the semantic vector associated with at least one word.

[0024] In the semantic element vector library of this embodiment, the association between words and semantic elements can be stored, and the corresponding relationship between semantic elements and semantic element vectors can also be stored. In the process of reading the semantic element vector associated with a word, the semantic element associated with the word is first read in the semantic element vector library, and then the semantic element vector corresponding to the semantic element is read; in addition, the association between words, semantic elements and semantic element vectors can also be directly stored, which is not limited in this embodiment. In order to improve the richness of the semantic element vector library and avoid the inability to read the semantic element vector associated with a word in the semantic element vector library during the semantic element vector extraction process, the semantic element vector library can also be generated for different business projects. In an optional implementation provided in this embodiment, the semantic element vector library can be generated in the following manner: Obtain project terms of business projects, perform word mining based on project terms to obtain related terms, and configure semantic element sequences for project terms and related terms; Input the contrasting sentences into the semantic component recognition model to perform semantic component recognition, and obtain the semantic component sequence of the different words in the contrasting sentences; Semantic element encoding is performed according to the semantic element sequence of each basic word in the basic vocabulary, the semantic element sequence of the project word, the semantic element sequence of the associated word and the semantic element sequence of the difference word, so as to obtain the semantic element vector corresponding to each semantic element; A semantic element vector library is generated according to the semantic element sequence of each word and the semantic element vector corresponding to each semantic element.

[0025] Optionally, the basic vocabulary may include vocabulary with the same or similar semantics (a graph relationship between every two words in the vocabulary becomes another word by adding and / or reducing a certain semantic element), a vocabulary with the same or similar part of speech (a graph relationship between every two words in the vocabulary becomes another word by adding and / or reducing a certain grammatical semantic element), and the basic vocabulary may be constructed according to the official dictionary of a certain language; this embodiment is not limited here.

[0026] The word mining includes searching for synonyms and / or antonyms of the project words as associated words. The comparison sentences include two pre-configured sentences that differ in only one word; wherein one of the comparison sentences is a legal sentence and the other is an illegal sentence. Legal sentences include sentences that can be understood normally, and illegal sentences include sentences that cannot be understood normally; for example, a legal sentence is "I love to eat fruit"; an illegal sentence is "I love to eat cement".

[0027] It should be noted that, in the present embodiment, model training can be performed using corpus data sets corresponding to multiple corpus categories, that is, it is necessary to filter the initial corpus data sets corresponding to the multiple corpus categories to obtain a corpus data set corresponding to each corpus category; in the present embodiment, the process of filtering the initial corpus data set corresponding to the first corpus category to obtain the corpus data set corresponding to the first corpus category is used as an example for explanation, and the process of filtering the initial corpus data sets corresponding to other corpus categories to obtain the corpus data sets is similar, which will not be repeated in this embodiment.

[0028] Step S204: determining at least one semantic element vector corresponding to the first corpus category according to the semantic element vector sequence, and calculating a first occurrence number of each semantic element vector in the first corpus category and a second occurrence number in the second corpus category.

[0029] After obtaining the semantic element vector sequence of each corpus data corresponding to the first corpus category as described above, in this step, starting from the semantic element vector sequence of each corpus data corresponding to the first corpus category, at least one semantic element vector corresponding to the first corpus category is determined. In addition, the first occurrence number of each semantic element vector corresponding to the first corpus category under the first corpus category and the second occurrence number under the second corpus category are calculated. Optionally, the second corpus category includes other corpus categories besides the first corpus category. The second occurrence number includes the occurrence number of the semantic element vector under other corpus categories; for example, it includes corpus category 1, corpus category 2 and corpus category 3; wherein the second occurrence number of the semantic element vector 1 corresponding to the first corpus category 1 under the second corpus category includes the occurrence number of the semantic element vector 1 under corpus category 2 and the occurrence number under corpus category 3.

[0030] In a specific implementation, the semantic element vector sequences of multiple corpus data may be first deduplicated to obtain at least one semantic element vector corresponding to the corpus category; for example, the initial corpus data set corresponding to corpus category 1 includes corpus data 1, corpus data 2, and corpus data 3; wherein the semantic element vector sequence of corpus data 1 is {{[semetic element vector 1], [semetic element vector 2]}, {[semetic element vector 1], [semetic element vector 3], [semetic element vector 4]}, {[semetic element vector 1], [semetic element vector 2], [semetic element vector 4]}}; The sequence of semantic vectors of corpus data 2 is {{[semetic vector 1], [semetic vector 4]}, {[semetic vector 1], [semetic vector 3], [semetic vector 4]}, {[semetic vector 1], [semetic vector 2], [semetic vector 5]}}; The sequence of semantic vectors of corpus data 3 is {{[semetic vector 3], [semetic vector 4]]}, {[semetic vector 1], [semetic vector 4], [semetic vector 5]}, {[semetic vector 1], [semetic vector 5]}}; De-duplicate the semantic element vectors in the semantic element vector sequences of corpus data 1, corpus data 2, and corpus data 3, and obtain semantic element vectors corresponding to corpus category 1, including [semantic element vector 1], [semantic element vector 2], [semantic element vector 3], [semantic element vector 4], and [semantic element vector 5]; Further, after determining at least one semantic element vector corresponding to the corpus category, the first occurrence number of each semantic element vector under the first corpus category and the second occurrence number under the second corpus category are calculated; in the process of calculating the first occurrence number of each semantic element vector, the occurrence number of the semantic element vector appearing independently can be calculated as the first occurrence number, and the occurrence number of the semantic element vector appearing independently can also be used as the first sub-occurrence number, and the simultaneous occurrence number of each semantic element vector appearing simultaneously with other semantic element vectors together constitutes the first occurrence number; that is, optionally, the first occurrence number includes the first sub-occurrence number of independent occurrences and the simultaneous occurrence number (combined occurrence number) of simultaneous occurrences (combined occurrences) with other semantic element vectors. Correspondingly, the second occurrence number includes the occurrence number of independent occurrences under the second corpus category.

[0031] Using the above example, semantic vector 1 appears 3 times in corpus data 1, 3 times in corpus data 2, and 2 times in corpus data 3. Then the first sub-occurrence number of semantic vector 1 is calculated, that is, the independent occurrence number is 3+3+2=8; similarly, the first sub-occurrence number of semantic vector 2 is calculated to be 3, the first sub-occurrence number of semantic vector 3 is 3, the first sub-occurrence number of semantic vector 4 is 6, and the first sub-occurrence number of semantic vector 5 is 3; in addition to calculating the first sub-occurrence number of each semantic vector, the simultaneous occurrence number of each semantic vector and other semantic vectors, or the combined occurrence number of combined appearances is also calculated. Semantic vector 1 and semantic vector 2 appear simultaneously 2 times in corpus data 1, 1 time in corpus data 2, and 0 times in corpus data 3. Then the simultaneous occurrence number of semantic vector 1 and semantic vector 2 is calculated, that is, the combined occurrence number is 2+1=3; similarly, the simultaneous occurrence number of semantic vector 1 and other semantic vectors and the simultaneous occurrence number between other semantic vectors are calculated.

[0032] The second occurrence number of each semantic element vector in the second corpus category is similar to the calculation process of the first sub-occurrence number of each semantic element vector in the first corpus category. Based on the semantic element vector sequence of each corpus data corresponding to the second corpus category, the second occurrence number of each semantic element vector in the first corpus category in the second corpus category is calculated, and this embodiment will not be repeated here.

[0033] Step S206: Determine the type of each sememe vector under the first corpus category according to the first occurrence number and the second occurrence number of each sememe vector.

[0034] After determining at least one semantic element vector corresponding to the corpus category based on the semantic element vector sequence and calculating the first occurrence number of each semantic element vector under the first corpus category and the second occurrence number of each semantic element vector under the second corpus category, the type of each semantic element vector under the first corpus category is determined based on the first occurrence number and the second occurrence number of each semantic element vector.

[0035] In a specific implementation, in order to improve the effectiveness of the type of the determined semantic element vector so that the type of the determined semantic element vector can accurately reflect the matching degree of the semantic element vector to the corpus category, in an optional implementation provided by this embodiment, in the process of determining the type of each semantic element vector under the first corpus category according to the at least one semantic element vector, the first number of occurrences and the second number of occurrences of each semantic element vector, the following operations are performed: (1) determining a first type of semantic element vector and a second type of semantic element vector under the first corpus category according to the first occurrence number of each semantic element vector; Optionally, the first type may be a core type, and the semantic element vector of the core type may be used as a core semantic element vector; wherein the semantic element vector of the core type may include a first preset number of semantic element vectors that appear most frequently under the first corpus category; the second type may be an auxiliary type, and the semantic element vector of the auxiliary type may be used as an auxiliary semantic element vector; wherein the semantic element vector of the auxiliary type may include a second preset number of semantic element vectors that appear most frequently at the same time as the core semantic element vector.

[0036] Generally speaking, the corpus data with more occurrences in the initial corpus data set corresponding to the first corpus category is more matched with the first corpus category. Similarly, the semantic element vector with more occurrences is also more matched with the first corpus category to a certain extent. The semantic element vector that appears at the same time as the semantic element vector with more occurrences not only represents the coordination of words, but also represents the corpus category to a certain extent. Therefore, in order to make the type of the determined semantic element vector more accurate and to retain the corpus data corresponding to the semantic element vector that appears more frequently in the initial corpus data set as much as possible during the filtering process, in this embodiment, the semantic element vector of the first type and the semantic element vector of the second type under the first corpus category can be first determined according to the first occurrence number of each semantic element vector; in an optional implementation manner provided by this embodiment, in the process of determining the semantic element vector of the first type and the semantic element vector of the second type under the first corpus category according to the first occurrence number of each semantic element vector, the following operations are performed: Arrange the at least one seme vector in descending order according to the number of occurrences of the first sub-eme vector of each seme vector to obtain a first seme vector queue; Determine the type of the semantic vectors before the first preset position in the first semantic vector queue as the first type; Arrange the remaining element vectors in descending order according to the number of occurrences of the second sub-elements of the remaining element vectors to obtain a second element vector queue; The type of the element vectors before the second preset position in the second element vector queue is determined as the second type.

[0037] Optionally, the second sub-occurrence number includes the number of occurrences simultaneously appearing with the first type of semantic element vector. The first preset rank and the second preset rank may be the same or different.

[0038] Specifically, a first preset number of semantic element vectors with the highest number of first sub-occurrences in at least one semantic element vector corresponding to the first corpus category are determined as semantic element vectors of the first type, and a second preset number of semantic element vectors with the highest number of simultaneous occurrences (i.e., the second sub-occurrences) with the first type of semantic element vector are determined as semantic element vectors of the second type. The first preset number corresponds to the first preset rank, and the second preset number corresponds to the second preset rank.

[0039] For example, if the first preset number is n, then the n semantic element vectors with the highest number of first sub-appearances in at least one semantic element vector are determined as the first type of semantic element vectors, that is, the core semantic element vectors; if the second preset number is m, then the n semantic element vectors with the highest number of appearances at the same time as the core semantic element vector are determined as the second type of semantic element vectors, that is, the auxiliary semantic element vectors.

[0040] It should be noted that, when there are multiple first-type semantic vectors, the sum of the number of simultaneous occurrences of any remaining semantic vector and each first-type semantic vector can be used as the second sub-occurrence number of any remaining semantic vector and the first-type semantic vector.

[0041] (2) Determine a third type of semantic element vector under the corpus category according to the first occurrence count and the second occurrence count of each semantic element vector.

[0042] Optionally, the third type may be a confusion type, and the semantic element vector of the confusion type may be used as a confusion semantic element vector; wherein the semantic element vector of the confusion type may include a semantic element vector whose number of occurrences (independent occurrences or first sub-occurrences) is close to that of the second corpus category.

[0043] In addition to determining the first type of semantic element vector and the second type of semantic element vector according to the first number of occurrences, if the number of occurrences of a semantic element vector under the first corpus category and the number of occurrences under the second corpus category are close, the semantic element vector cannot clearly represent the first corpus category and the second corpus category. Therefore, this semantic element vector can be determined as the third type to filter out the corpus data corresponding to the semantic element vector during the filtering process; based on this, in an optional implementation manner provided by this embodiment, the process of determining the third type of semantic element vector under the corpus category according to the first number of occurrences and the second number of occurrences of each semantic element vector is implemented in the following manner: Calculate the difference between the first sub-appearance number and the second sub-appearance number of each element vector; The type of the sememe vector whose occurrence count difference is less than the difference threshold is determined as the third type.

[0044] Specifically, a semantic element vector in at least one semantic element vector corresponding to the first corpus category whose difference in occurrence number between that in the second corpus category and that in the second corpus category is less than a difference threshold may be determined as a semantic element vector of the third type, i.e., a confused semantic element vector.

[0045] It should be noted that, since the second corpus category may be multiple, the second occurrence count of each semantic element vector is also multiple. Therefore, if the difference between the first sub-occurrence count of a semantic element vector and any second occurrence count is less than the difference threshold, the type of the semantic element vector is determined to be the third type.

[0046] In addition, the type of each semantic element vector under the first corpus category can also be determined by the type recognition model; for example, at least one semantic element vector, the first occurrence number and the second occurrence number of each semantic element vector can be input into the type recognition model for type recognition to obtain the type of each semantic element vector. The type recognition model can be a linear model, a neural network, or other structures, which is not limited in this embodiment. Specifically, after obtaining the semantic element vector sequence of each corpus data, at least one semantic element vector corresponding to the first corpus category can be determined according to the semantic element vector sequence, and the first occurrence number of each semantic element vector under the first corpus category and the second occurrence number of each semantic element vector under the second corpus category are calculated, and then at least one semantic element vector, the first occurrence number and the second occurrence number of each semantic element vector are input into the type recognition model for type recognition to obtain the type of each semantic element vector. In addition, after obtaining the semantic element vector sequence of each corpus data, the semantic element vector sequence of each corpus data can be input into the type recognition model for type recognition to obtain the type of each semantic element vector. In the process of type recognition by the type recognition model, first, according to the semantic vector sequence, at least one semantic vector corresponding to the corpus category and the first sub-occurrence number of each semantic vector are determined, and the semantic vector of the first type is determined in at least one semantic vector according to the first sub-occurrence number of each semantic vector, and the remaining semantic vectors and the second sub-occurrence number of the semantic vector of the first type are calculated, and the semantic vector of the second type is determined in the remaining semantic vector according to the second sub-occurrence number of the remaining semantic vectors. In addition, the semantic vector whose difference in the number of occurrences with the second number of occurrences under the second corpus category is less than the difference threshold is determined as the semantic vector of the third type. Specifically, the process of determining the type of each semantic vector by the type recognition model is similar to the above, and the above-mentioned related content can be referred to, and this embodiment will not be repeated here.

[0047] It should also be noted that in the process of determining the type of each semantic element vector using the above method, there may be a semantic element vector that is determined as both the first type and the third type, or a semantic element vector that is determined as both the second type and the third type. In this case, the semantic element vector is determined as the third type.

[0048] Step S208: filtering the initial corpus data set according to the type to obtain a corpus data set.

[0049] After obtaining the type of each semantic element vector, in order to achieve effective filtering of the initial corpus data set, the corpus data set is filtered according to the type to obtain the corpus data set.

[0050] In specific implementation, in order to avoid the situation where the corpus data set corresponding to the first corpus category contains too little corpus data due to the small amount of corpus data in the initial corpus data set and the corpus data set being directly filtered, this embodiment provides an optional implementation mode, in which the data proportion of the first corpus category can also be calculated based on the number of corpus data in the initial corpus data set and the total number of corpus data; it is detected whether the data proportion of the first corpus category is greater than a pre-configured proportion threshold of the first corpus category; if so, the initial corpus data set is filtered according to the type to obtain the corpus data set; if not, no processing is performed.

[0051] In the process of filtering the initial corpus data set according to the type, the number of remaining data can also be calculated according to the total number of corpus data and the proportion threshold, and then the initial corpus data set is filtered according to the number of remaining data and the type of each semantic vector to obtain a corpus data set consisting of the corpus data of the remaining number of data. Optionally, the total number of corpus data is the sum of the number of corpus data in the initial corpus data set corresponding to each corpus category in the multiple corpus categories, and the number of remaining data is the number of corpus data that should be included in the corpus data set of the first corpus category.

[0052] In the specific implementation process, in order to filter out the corpus data with a low matching degree with the first corpus category during the filtering process and ensure that the corpus data retained in the corpus data set has a high matching degree with the first corpus category, in an optional implementation provided by this embodiment, in the process of filtering the initial corpus data set according to the type to obtain the corpus data set, the corpus data to be deleted in the initial corpus data set can be first determined according to the type, and then the corpus data to be deleted is deleted from the initial corpus data set to obtain the corpus data set; Furthermore, in an optional implementation manner provided by the present embodiment, in the process of determining the corpus data to be deleted in the initial corpus data according to the type, the first corpus data that does not contain the first type of semantic element vector in the initial corpus data set can be read first, and the second corpus data that contains the third type of semantic element vector in the initial corpus data set can be read; and then the corpus data in the first corpus data and the second corpus data that contains the second type of semantic element vector whose number is less than the number threshold is determined as the corpus data to be deleted. In addition, the corpus data to be deleted in the initial corpus data set can be determined first according to the type and the number of remaining data, and then the corpus data to be deleted is deleted from the initial corpus data set to obtain the corpus data set, that is, the corpus data to be deleted is determined in the first corpus data and the second corpus data according to the number of remaining data and the number of semantic element vectors of the second type contained.

[0053] For example, the corpus data that does not contain the first type of semantic element vectors in the initial corpus data set is determined as the first corpus data, the corpus data that contains the third type of semantic element vectors in the initial corpus data set is determined as the second corpus data, and the corpus data other than the remaining number of corpus data containing the largest number of second type semantic element vectors in the first corpus data and the second corpus data is determined as the corpus data to be deleted; that is, after determining the first corpus data and the second corpus data, the first corpus data and the second corpus data are arranged in descending order according to the number of second type semantic element vectors contained to obtain a corpus data queue, and the corpus data after the corresponding position of the remaining number of data in the corpus data queue is determined as the corpus data to be deleted.

[0054] In addition, after determining the first corpus data and the second corpus data, the first corpus data and the second corpus data can be directly determined as the first corpus data to be deleted, and the remaining corpus data (corpus data other than the first corpus data and the second corpus data in the initial corpus data set) containing the second type of semantic element vectors whose number is less than a preset threshold can be determined as the second corpus data to be deleted; or the second corpus data to be deleted can be determined from the remaining corpus data according to the number of remaining data, that is, the corpus data other than the number of remaining data corpus data containing the largest number of semantic element vectors in the remaining corpus data can be determined as the second corpus data to be deleted, and then the first corpus data to be deleted and the second corpus data to be deleted can be determined as the corpus data to be deleted.

[0055] In addition, the configuration of the deleted semantic element vector can be performed in advance for the first corpus category. In the process of filtering the initial corpus data set, the initial corpus data set is first filtered according to the deleted semantic element vector configured for the first corpus category to obtain an intermediate corpus data set, and then the intermediate corpus data set is filtered for a second time according to the type to obtain the corpus data set. It should be noted that if the number of remaining data is introduced, a data proportion detection can be performed before filtering. Specifically, if the data proportion of the first corpus category is greater than the proportion threshold of the first corpus category, the initial corpus data set is filtered according to the deleted semantic element vector configured for the corpus category to obtain the intermediate corpus data set, that is, the corpus data containing the deleted semantic element vector in the initial corpus data set is deleted to obtain the intermediate corpus data set. After obtaining the intermediate corpus data set, the data proportion detection is performed again. If the data proportion of the intermediate corpus data set is greater than the proportion threshold, the intermediate corpus data set is filtered for a second time according to one of the types or both the type and the number of remaining data to obtain the corpus data set.

[0056] The above describes the process of filtering the number of remaining data calculated according to the total number of corpus data and the percentage threshold. In addition, in this embodiment, if the data percentage of the initial corpus data set of the first corpus category is greater than the percentage threshold of the first corpus category, the initial corpus data set is first filtered according to the pre-configured filtering strategy to obtain an intermediate corpus data set, and then the intermediate corpus data set is filtered for a second time according to one type or both the type and the number of remaining data to obtain a corpus data set. In the process of filtering the initial corpus data set according to the pre-configured filtering strategy, filtering can be performed according to the length of the corpus data, sentence pattern, and sentence vector distance, so that the corpus data in the intermediate corpus data set have differences in the length of the corpus data, rich sentence patterns, and large sentence vector distance.

[0057] The above takes the first corpus category as an example to illustrate the process of filtering the initial corpus data set. In this embodiment, the initial corpus data sets corresponding to multiple corpus categories can be filtered in the above manner to obtain the corpus data sets corresponding to each corpus category, and then the model training is performed according to the corpus data sets of multiple corpus categories. Optionally, corpus data sets corresponding to multiple corpus categories can be obtained, and the intention recognition model can be trained according to the corpus data sets corresponding to multiple corpus categories and each corpus category. In addition, state recognition models, identity recognition models, preference recognition models, etc. can also be trained according to the corpus data sets of multiple corpus categories, and other models can also be trained, which are not limited in this embodiment. And the model to be trained can be a neural network structure, a linear model or other structures, which are not limited in this embodiment. It should also be noted that the semantic vector model and the type recognition model in this embodiment can also be a large language model. If it is a large language model, in addition to inputting relevant features, the corresponding prompt text is also input.

[0058] The following describes the filtering process and model training process of the initial corpus data sets of multiple corpus categories. In specific implementation, the corpus data set to be filtered can be first obtained; optionally, the corpus data set to be filtered includes an initial corpus data set corresponding to each corpus category in at least one corpus category; here, the initial corpus data set can be a subset; after obtaining the corpus data set to be filtered, semantic vectors can be extracted for each corpus data in the corpus data set to be filtered to obtain a semantic vector sequence for each corpus data, or each corpus data in the corpus data set to be filtered can be input into a semantic vector model for semantic vector extraction to obtain a semantic vector sequence for each corpus data, and then, based on the semantic vector sequence of at least one corpus data corresponding to each corpus category, at least one semantic vector corresponding to each corpus category is determined, and each semantic vector is calculated in Corresponding to the first occurrence number under the first corpus category and the second occurrence number under the second corpus category, the type of each semantic vector under the corresponding first corpus category is determined according to the first occurrence number and the second occurrence number of each semantic vector corresponding to each corpus category, or at least one semantic vector and the first occurrence number and the second occurrence number of each semantic vector are input into the type recognition model for type recognition to obtain the type of each semantic vector under each corpus category, and then according to the type of each semantic vector under each corpus category, the corpus data in the corpus data set to be filtered are filtered to obtain the training corpus data set; optionally, the training corpus data set includes the corpus data set corresponding to each corpus category in at least one corpus category. Alternatively, in the process of determining the type of each semantic vector under each corpus category, the semantic vector sequence of each corpus category in the corpus data to be filtered and each corpus data corresponding to each corpus category can also be input into the type recognition model for type recognition to obtain the type of each semantic vector under each corpus category, which is not limited in this embodiment. It should also be noted that, during the process of type identification, the type identification model determines the first type of semantic element vector, the second type of semantic element vector and the third type of semantic element vector for each corpus category.

[0059] It should be noted that in order to improve the perception of each corpus category in the corpus data set to be filtered, in an optional implementation provided by this embodiment, after obtaining the corpus data set to be filtered, a corpus tree is generated based on the corpus data set to be filtered, so that in the subsequent filtering process of the corpus data in the corpus data set to be filtered, the corpus tree is pruned, so that other nodes in the corpus tree are synchronously updated through pruning during the filtering process, thereby improving the perception of the pruning result, that is, the filtering result. In the specific process of generating the corpus tree, the distribution parameters of each corpus category can be calculated first according to the corpus data set to be filtered and the initial corpus data set corresponding to each corpus category in the corpus data set to be filtered, and then the corpus tree is generated according to the corpus category and the distribution parameters.

[0060] For example, in the robot agent scenario, the corpus category includes user voice texts, and the user voice data includes 8,000 items in total, of which the user's reply voice texts to the robot agent's verification include 200 affirmative intentions and 170 negative intentions; under the affirmative intention branch of verification, the user's promise to return resources in response to the robot agent's inquiry whether to return the resources is 50, and the intention of failing to return the resources is 70; the distribution parameters of the affirmative intention are calculated to be 2.5% (200 / 8000), the distribution parameter of the negative intention is 2.2125% (170 / 8000), the distribution parameter of the intention to promise to return resources is 0.0625 (50 / 8000), and the distribution parameter of the intention to fail to return resources is 0.0875 (70 / 8000); then it can generate, for example Figure 3 The corpus tree shown in Figure 1 is shown in Figure 2. Figure 3 The corpus tree shown only shows part of the corpus tree, and may also include other branches, which are not limited in this embodiment. In the process of generating the corpus tree, the corpus tree may also be generated according to the corpus category, distribution parameters and pre-configured business logic data. For example, the pre-configured business logic data includes multiple branches under positive intent, and does not include branches under negative intent.

[0061] After obtaining a corpus data set of each corpus category in multiple corpus categories, multiple experimental groups are determined from the multiple corpus data sets, and model training is performed based on each experimental group, and the model with the best performance is used as the intent recognition model under the corresponding business project. Specifically, in the process of determining multiple experimental groups from multiple corpus data sets, corpus data can be randomly extracted from the corpus data set corresponding to each corpus category to obtain training corpus data, and the training corpus data is used as a training sample, and the corresponding corpus category is used as a sample label to train the intent recognition model.

[0062] On the basis of the intention recognition model obtained through training, in order to further improve the intention recognition effect of the intention recognition model, after obtaining the user voice data for intention recognition based on the intention recognition model in real time and the user intention obtained by the intention recognition model, the user voice data can be used as the corpus data, and the corresponding user intention can be used as the corpus category. The above filtering process is repeated to obtain the real-time corpus data set corresponding to each corpus category, and then the real-time intention recognition model is trained by the real-time corpus data set. If the performance of the real-time intention recognition model is better than the above intention recognition model, the real-time intention recognition model is used as the intention recognition model under the corresponding business project. If the performance of the real-time intention recognition model is not better than the above recognition model, the above intention recognition model is still used for the intention recognition under the corresponding business project. In this way, in the actual use process, the model can be retrained or the intention recognition model can be trained twice through real-time user voice data and user intention to improve the performance of the intention recognition model. In addition, a real-time corpus tree can be generated based on the user voice data and user intention and sent to the service user, so that the service user can adjust the content of the robot broadcast according to the real-time corpus tree.

[0063] In order to avoid large differences in the distribution of user voice data and user intent due to the particularity of intent recognition time, after obtaining the user voice data and user intent, it is possible to detect whether the intent recognition time meets the secondary training conditions. If so, the user voice data is used as corpus data, and the corresponding user intent is used as the corpus category, and the above filtering process is repeated; if not, no processing is performed.

[0064] In summary, the data processing method provided in this embodiment extracts semantic vectors from each corpus data in the initial corpus data set corresponding to the first corpus category to obtain a semantic vector sequence for each corpus data, and determines at least one semantic vector corresponding to the first corpus category according to the semantic vector sequence. According to the semantic vector sequence, the first preset number of semantic vectors with the largest number of occurrences under the first corpus category are determined as the first type, the second preset number of semantic vectors with the largest number of occurrences under the first corpus category and the first type of semantic vectors are determined as the second type, and the semantic vectors whose difference in the number of occurrences between the first corpus category and the second corpus category is less than a difference threshold are determined as the third type. Then, the corpus data that does not contain the semantic vector of the first type in the initial corpus data and the corpus data that contains the semantic vector of the third type, whose number of semantic vectors of the second type is less than the number threshold, are deleted. In this way, since the semantic vector of the first type appears the most times in the initial corpus database, the semantic vector of the first type is The semantic vector of the third type is most matched with the corpus category, and the semantic vector of the second type is second. Since the difference between the number of occurrences of the semantic vector of the third type in the first corpus category and the number of occurrences in the second category is small, it is impossible to determine whether the semantic vector is more matched with the first corpus category or the second corpus category. Therefore, the semantic vector is determined as the third type and deleted in the filtering process. In this way, a method for determining the type of semantic vector is proposed. By determining the type of each semantic vector based on the initial corpus data set, it is adaptive to different projects of different businesses, making the filtering processing of the initial corpus data set more directional, and the corpus data corresponding to the semantic vector with a low matching degree with the corpus category can be filtered, thereby improving the effectiveness of filtering, and also improving the accuracy and effectiveness of the corpus data in the obtained corpus data set. In addition, a corpus category proportion threshold is introduced to make the number of corpus data sets obtained after filtering more accurate with the corresponding business, further improving the performance of the model trained using the corpus data set in the future.

[0065] The following takes the application of a data processing method provided in this embodiment in a model training scenario as an example to further illustrate the data processing method provided in this embodiment for use in a model training scenario. Figure 4 , the data processing method applied to the model training scenario specifically includes the following steps.

[0066] Step S402: obtaining a corpus data set to be filtered for a business project.

[0067] Optionally, the corpus data set to be filtered includes at least one corpus category and an initial corpus data subset corresponding to each corpus category.

[0068] Step S404: Generate an initial corpus tree according to the pre-configured corpus tree generation strategy of the business project and the corpus data set to be filtered.

[0069] Step S406: Input each corpus data in the corpus data set to be filtered into the semantic component vector model to extract the semantic component vector, and obtain the semantic component vector sequence of each corpus data.

[0070] Step S408: Determine at least one semantic element vector corresponding to each corpus category according to the semantic element vector sequence of at least one corpus data corresponding to each corpus category.

[0071] Step S410, calculating the first occurrence number of each sememe vector in the corresponding first corpus category and the second occurrence number in the second corpus category.

[0072] Step S412: determining the type of each sememe vector under the corresponding first corpus category according to the first occurrence number and the second occurrence number of at least one sememe vector corresponding to each corpus category.

[0073] In addition, step S410 and step S412 can also be replaced by: inputting at least one semantic element vector corresponding to each corpus category and the semantic element vector sequence of each corpus data into the type recognition model for type recognition to obtain the type of each semantic element vector under each corpus category.

[0074] Step S414: selecting a corpus category to be pruned from at least one corpus category according to the data proportion and proportion threshold of each corpus category.

[0075] In addition, step S414 can also be executed after step S404. Accordingly, the corpus category to be pruned is processed in steps S406 to S410. For example, the corpus data corresponding to the corpus category to be pruned in the corpus data set to be filtered is input into the semantic vector model for semantic vector extraction to obtain a semantic vector sequence of each corpus data under the corpus category to be pruned. According to the semantic vector sequence of each corpus data under the corpus category to be pruned, at least one semantic vector corresponding to the corpus category to be pruned is determined, and the first occurrence number of each semantic vector under the corresponding first corpus category and the second occurrence number under the second corpus category are calculated. At least one semantic vector and the first occurrence number and the second occurrence number of each semantic vector are input into the type recognition model for type recognition to obtain the type of each semantic vector under the corpus category to be pruned.

[0076] Step S416, determining the number of remaining data corresponding to the corpus category to be pruned according to the total number of corpus data and the percentage threshold of the corpus category to be pruned.

[0077] Step S418, according to the type of each semantic element vector under the corpus category to be pruned and the corresponding number of remaining data, prune the corpus data corresponding to the corpus category to be pruned in the initial corpus tree to obtain a corpus tree.

[0078] Step S420, performing model training according to the corpus data in the corpus tree to obtain an intent recognition model under the business project.

[0079] It should be noted that any one of steps S402 to S420 or any combination of multiple steps can be combined with any one of steps S202 to S208 to form a new implementation method according to the needs of implementation deployment; in addition, according to the needs of actual deployment, any one or multiple technical features can be selected in steps S402 to S420 and combined with any one or multiple technical features provided in steps S202 to S208 to form a new implementation method; or, any one or multiple technical features in steps S402 to S420 can be replaced with any one or multiple technical features provided in steps S202 to S208 to form a new implementation method according to the needs of actual deployment, which will not be repeated here.

[0080] An embodiment of a data processing device provided in this specification is as follows: In the above-mentioned embodiment, a data processing method is provided, and correspondingly, a data processing device is also provided, which will be described below with reference to the accompanying drawings.

[0081] Reference Figure 5 , which shows a schematic diagram of the structure of a data processing device provided by this embodiment. Since the device embodiment corresponds to the method embodiment, the description is relatively simple, and the relevant parts can refer to the corresponding description of the method embodiment provided above. The device embodiment described below is only illustrative.

[0082] This embodiment provides a data processing device, including: A semantic element vector extraction module 502 is used to obtain an initial corpus data set corresponding to the first corpus category, and extract semantic element vectors from each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data; A semantic element vector determination module 504, configured to determine at least one semantic element vector corresponding to the first corpus category according to the semantic element vector sequence, and calculate a first occurrence number of each semantic element vector in the first corpus category and a second occurrence number in the second corpus category; A type determination module 506, configured to determine the type of each sememe vector under the first corpus category according to the first number of occurrences and the second number of occurrences of each sememe vector; The filtering processing module 508 is used to filter the initial corpus data set according to the type to obtain a corpus data set.

[0083] In the data processing device provided in this embodiment, during the data processing process, for the initial corpus data set corresponding to the first corpus category, firstly, a semantic element vector is extracted for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data, and then at least one semantic element vector corresponding to the first corpus category is determined according to the semantic element vector sequence, and the first occurrence number of each semantic element vector under the first corpus category and the second occurrence number of each semantic element vector under the second corpus category are calculated, and then the type of each semantic element vector under the first corpus category is determined according to the first occurrence number and the second occurrence number of each semantic element vector; after obtaining at least one semantic element vector corresponding to the first corpus category, combined with the semantic element vector sequence corresponding to the first corpus category, the semantic element vector is extracted. The type of each semantic element vector in the first corpus category is determined based on the first occurrence number of each semantic element vector in the first corpus category and the second occurrence number of each semantic element vector corresponding to the first corpus category in the second corpus category, thereby improving the accuracy of the type of each determined semantic element vector in the first corpus category, and then filtering the initial corpus data set according to the type of each semantic element vector to obtain a corpus data set. In this way, the corpus data is filtered based on the semantic element vector. On the basis of realizing the filtering process of the initial corpus data set, it is ensured that the corpus data in the obtained corpus data set is more matched with the first corpus category, thereby improving the accuracy and effectiveness of the corpus data in the obtained corpus data set.

[0084] An embodiment of a data processing device provided in this specification is as follows: Corresponding to the data processing method described above, based on the same technical concept, an embodiment of the present application further provides a data processing device, which is used to execute the data processing method provided above. Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application.

[0085] This embodiment provides a data processing device, including: like Figure 6As shown, the data processing device may have relatively large differences due to different configurations or performances, and may include one or more processors 601 and memory 602, and the memory 602 may store one or more storage applications or data. Among them, the memory 602 may be a short-term storage or a persistent storage. The application stored in the memory 602 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the data processing device. Furthermore, the processor 601 may be configured to communicate with the memory 602 and execute a series of computer executable instructions in the memory 602 on the data processing device. The data processing device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input / output interfaces 605, one or more keyboards 606, etc.

[0086] In a specific embodiment, a data processing device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the data processing device, and the one or more programs configured to be executed by one or more processors include computer-executable instructions for performing the following: Acquire an initial corpus data set corresponding to the first corpus category, and extract semantic element vectors for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data; Determine at least one semantic element vector corresponding to the first corpus category according to the semantic element vector sequence, and calculate a first occurrence number of each semantic element vector in the first corpus category and a second occurrence number in the second corpus category; Determine the type of each sememe vector under the first corpus category according to the first number of occurrences and the second number of occurrences of each sememe vector; The initial corpus data set is filtered according to the type to obtain a corpus data set.

[0087] In the data processing device provided in the present embodiment, for an initial corpus data set corresponding to a first corpus category, firstly, semantic element vectors are extracted for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data, and then at least one semantic element vector corresponding to the first corpus category is determined according to the semantic element vector sequence, and the first occurrence number of each semantic element vector under the first corpus category and the second occurrence number of each semantic element vector under the second corpus category are calculated, and then the type of each semantic element vector under the first corpus category is determined according to the first occurrence number and the second occurrence number of each semantic element vector; after obtaining at least one semantic element vector corresponding to the first corpus category, in combination with each semantic element vector corresponding to the first corpus category, The type of each semantic element vector in the first corpus category is determined based on the first occurrence number of the semantic element vector in the first corpus category and the second occurrence number of each semantic element vector corresponding to the first corpus category in the second corpus category, so as to improve the accuracy of the type of each determined semantic element vector in the first corpus category, and then the initial corpus data set is filtered according to the type of each semantic element vector to obtain the corpus data set. In this way, the corpus data is filtered based on the semantic element vector. On the basis of realizing the filtering process of the initial corpus data set, it is ensured that the corpus data in the obtained corpus data set is more matched with the first corpus category, so as to improve the accuracy and effectiveness of the corpus data in the obtained corpus data set.

[0088] An embodiment of a computer-readable storage medium provided in this specification is as follows: Corresponding to the data processing method described above, based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium.

[0089] The computer-readable storage medium provided in this embodiment is used to store computer-executable instructions, and the computer-executable instructions implement the following process when executed: Acquire an initial corpus data set corresponding to the first corpus category, and extract semantic element vectors for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data; Determine at least one semantic element vector corresponding to the first corpus category according to the semantic element vector sequence, and calculate a first occurrence number of each semantic element vector in the first corpus category and a second occurrence number in the second corpus category; Determine the type of each sememe vector under the first corpus category according to the first number of occurrences and the second number of occurrences of each sememe vector; The initial corpus data set is filtered according to the type to obtain a corpus data set.

[0090] In the computer-readable storage medium provided in the present embodiment, for an initial corpus data set corresponding to a first corpus category, firstly, a semantic element vector is extracted for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data, and then at least one semantic element vector corresponding to the first corpus category is determined according to the semantic element vector sequence, and a first occurrence number of each semantic element vector under the first corpus category and a second occurrence number under the second corpus category are calculated, and then the type of each semantic element vector under the first corpus category is determined according to the first occurrence number and the second occurrence number of each semantic element vector; after obtaining at least one semantic element vector corresponding to the first corpus category, in combination with each semantic element vector corresponding to the first corpus category, The type of each semantic element vector in the first corpus category is determined based on the first occurrence number of the semantic element vector in the first corpus category and the second occurrence number of each semantic element vector corresponding to the first corpus category in the second corpus category, so as to improve the accuracy of the type of each determined semantic element vector in the first corpus category, and then filter the initial corpus data set according to the type of each semantic element vector to obtain the corpus data set. In this way, the corpus data is filtered based on the semantic element vector. On the basis of realizing the filtering process of the initial corpus data set, it is ensured that the corpus data in the obtained corpus data set is more matched with the first corpus category, so as to improve the accuracy and effectiveness of the corpus data in the obtained corpus data set.

[0091] It should be noted that an embodiment of a computer-readable storage medium in this specification and an embodiment of a data processing method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned corresponding method, and the repeated parts will not be repeated.

[0092] Another embodiment of the present disclosure further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following process is implemented: Acquire an initial corpus data set corresponding to the first corpus category, and extract semantic element vectors for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data; Determine at least one semantic element vector corresponding to the first corpus category according to the semantic element vector sequence, and calculate a first occurrence number of each semantic element vector in the first corpus category and a second occurrence number in the second corpus category; Determine the type of each sememe vector under the first corpus category according to the first number of occurrences and the second number of occurrences of each sememe vector; The initial corpus data set is filtered according to the type to obtain a corpus data set.

[0093] The computer program product in the embodiments of the present disclosure can implement the various processes of the above-mentioned data processing method embodiments and achieve the same effects and functions, which will not be repeated here.

[0094] In the computer program product provided in this embodiment, for an initial corpus data set corresponding to a first corpus category, firstly, a semantic element vector is extracted for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data, and then at least one semantic element vector corresponding to the first corpus category is determined according to the semantic element vector sequence, and a first occurrence number of each semantic element vector under the first corpus category and a second occurrence number of each semantic element vector under the second corpus category are calculated, and then the type of each semantic element vector under the first corpus category is determined according to the first occurrence number and the second occurrence number of each semantic element vector; after obtaining at least one semantic element vector corresponding to the first corpus category, in combination with each semantic element vector corresponding to the first corpus category, The type of each semantic element vector in the first corpus category is determined based on the first occurrence number of the vector in the first corpus category and the second occurrence number of each semantic element vector corresponding to the first corpus category in the second corpus category, thereby improving the accuracy of the type of each determined semantic element vector in the first corpus category, and then filtering the initial corpus data set according to the type of each semantic element vector to obtain the corpus data set. In this way, the corpus data is filtered based on the semantic element vector. On the basis of realizing the filtering process of the initial corpus data set, it is ensured that the corpus data in the obtained corpus data set is more matched with the first corpus category, thereby improving the accuracy and effectiveness of the corpus data in the obtained corpus data set.

[0095] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0096] It should be understood by those skilled in the art that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the embodiments of the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable test equipment to produce a machine, so that the instructions executed by the processor of the computer or other programmable test equipment generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0098] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable test equipment to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, the instruction device being implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable test device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0100] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0101] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0102] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0103] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0104] Embodiments of the present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0105] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0106] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire an initial corpus data set corresponding to the first corpus category, and extract semantic element vectors for each corpus data in the initial corpus data set to obtain a semantic element vector sequence for each corpus data; Determine at least one semantic element vector corresponding to the first corpus category according to the semantic element vector sequence, and calculate a first occurrence number of each semantic element vector in the first corpus category and a second occurrence number in the second corpus category; Determine the type of each sememe vector under the first corpus category according to the first number of occurrences and the second number of occurrences of each sememe vector; The initial corpus data set is filtered according to the type to obtain a corpus data set.

2. The method according to claim 1, characterized in that The extracting of semantic element vectors from each corpus data in the initial corpus data set to obtain a semantic element vector sequence of each corpus data includes: Performing word recognition on each corpus data to obtain at least one word contained in each corpus data; The semantic element vector associated with each word is read from the semantic element vector library, and a semantic element vector sequence of each corpus data is generated according to the semantic element vector associated with at least one word.

3. The method according to claim 1, characterized in that The determining, according to the first number of occurrences and the second number of occurrences of each sememe vector, the type of each sememe vector under the first corpus category comprises: Determine, according to the first occurrence number of each sememe vector, a first type of sememe vector and a second type of sememe vector under the first corpus category; Determine a third type of semantic element vector under the first corpus category according to the first occurrence number and the second occurrence number of each semantic element vector.

4. The method according to claim 3, characterized in that The determining, according to the first occurrence number of each sememe vector, a first type of sememe vector and a second type of sememe vector under the first corpus category comprises: Arrange the at least one seme vector in descending order according to the first sub-occurrence number of each seme vector to obtain a first seme vector queue; the first sub-occurrence number includes the number of independent occurrences of the seme vector; Determine the type of the semantic vectors before the first preset position in the first semantic vector queue as the first type; Arrange the remaining element vectors in descending order according to the second sub-occurrence number of the remaining element vectors to obtain a second element vector queue; the second sub-occurrence number includes the number of times the remaining element vector and the first type of element vector appear at the same time; The type of the element vectors before the second preset position in the second element vector queue is determined as the second type.

5. The method according to claim 3, characterized in that: Determining the third type of semantic element vector under the corpus category according to the first number of occurrences and the second number of occurrences of each semantic element vector includes: Calculate the difference between the first sub-appearance number and the second sub-appearance number of each element vector; The type of the sememe vector whose occurrence count difference is less than the difference threshold is determined as the third type.

6. The method according to claim 1, characterized in that The filtering process of the initial corpus data set according to the type to obtain the corpus data set includes: According to the type, determining the corpus data to be deleted in the initial corpus data set; The corpus data to be deleted is deleted from the initial corpus data set to obtain the corpus data set.

7. The method according to claim 5, characterized in that The step of determining the corpus data to be deleted in the initial corpus data set according to the type includes: Reading first corpus data that does not include the first type of semantic element vectors in the initial corpus data set, and reading second corpus data that includes the third type of semantic element vectors in the initial corpus data set; The corpus data including the second type of semantic element vector number less than the number threshold in the first corpus data and the second corpus data is determined as the corpus data to be deleted.

8. A data processing device, characterized in that: The device comprises: A processor; and a memory configured to store computer executable instructions, which, when executed, cause the processor to perform the data processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the data processing method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the data processing method according to any one of claims 1 to 7.

Images