Data processing method, device and equipment, computer readable storage medium and computer program product

Abstract

The invention provides a data processing method, device and equipment, a computer readable storage medium and a computer program product. The method comprises the steps of performing word segmentation processing on a first text to obtain different first words; determining the importance degree of the first word based on the occurrence frequency of the first word in the first text; querying a context word of the first word in a first text comprising the first word; based on the occurrence frequency of the context word of the first word, determining a conditional entropy of the context word corresponding to the first word; and based on the context window, the number of the first words, the conditional entropy of the context words corresponding to the first words, and the importance of the first words, determining the word vector dimension of the first text. The word vector dimension determined by the method and the device is adaptive to the word distribution condition in the first text, so that the first text can be represented more accurately.

Description

Technical Field

[0001] The present application relates to data processing technology, and in particular to a data processing method, apparatus, device, computer-readable storage medium, and computer program product. Background Art

[0002] The dimension of a word vector refers to the number of values ​​contained in a vector when mapping a word to a word. It is a key parameter in word vector models. High-dimensional word vectors can capture richer semantic and grammatical features. For example, word vectors can reflect semantic similarity through distance in a high-dimensional space. However, too high a dimension of a word vector increases computational complexity and may lead to overfitting. On the other hand, too low a dimension cannot fully represent the complexity of the data and may lose important information.

[0003] In the related technology, D≈logN is used for approximate estimation, where N represents the total number of words and D is an integer representing the word vector dimension. This scheme takes into account that the relationship between words grows exponentially with the vocabulary size, and the logarithmic relationship can better balance the expressive power and computational cost, that is, when N increases, logN grows relatively slowly, so that a certain expressive power can be maintained without excessively increasing the dimension. However, the related technology only considers the total number of words. If there are two sources of text collections, each text collection contains N words, but the N words in one text collection are the same, and the N words in the other text collection are different, according to the existing scheme, the same dimensional word vectors are used to represent the two text collections, which is obviously unreasonable. This single determination method ignores the internal word distribution characteristics of the text collection, and the determined word vector dimension brings a bottleneck to the accuracy of the word vector representation. Summary of the Invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device, computer-readable storage medium, and computer program product. The word vector dimension determined by the present application is adapted to the word distribution in the first text, and the first text can be more accurately represented.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] This embodiment of the present application provides a data processing method, the method comprising:

[0007] Performing word segmentation processing on the plurality of first texts to obtain a plurality of mutually different first words;

[0008] For each of the first words, determining the importance of the first word based on the number of occurrences of the first word in the plurality of first texts;

[0009] For each of the first words, searching a first text including the first word for a context word of the first word, wherein in the first text, the context word of the first word is a word within a context window of the first word in the first text;

[0010] For each of the first words, determining a conditional entropy of the first word corresponding to the context word based on the number of occurrences of the context word of the first word;

[0011] The word vector dimensions of the plurality of first texts are determined based on the context window, the number of the first words, the conditional entropy of each of the first words corresponding to the context words, and the importance of each of the first words.

[0012] An embodiment of the present application provides a data processing device, including:

[0013] A word segmentation module, configured to perform word segmentation processing on the plurality of first texts to obtain a plurality of different first words;

[0014] an importance determination module, configured to determine, for each of the first words, the importance of the first word based on the number of occurrences of the first word in the plurality of first texts;

[0015] a context word determination module, configured to, for each first word, search a first text containing the first word for a context word of the first word, wherein in the first text, the context word of the first word is a word within a context window of the first word in the first text;

[0016] a conditional entropy determination module, configured to determine, for each first word, a conditional entropy of the first word corresponding to the context word based on the number of occurrences of the context word of the first word;

[0017] A word vector dimension determination module is used to determine the word vector dimension of the multiple first texts based on the context window, the number of the first words, the conditional entropy of each first word corresponding to the context word, and the importance of each first word.

[0018] In the above scheme, the word segmentation module is also used to perform word segmentation processing on the multiple first texts to obtain second words included in the multiple first texts; and perform deduplication processing on the second words included in the multiple first texts to obtain multiple different first words.

[0019] In the above scheme, the importance determination module is also used to determine a first ratio of the total number of words in the multiple first texts to the number of times the first word appears in the multiple first texts; perform logarithmic processing on the first ratio to obtain a first logarithmic result; based on the first logarithmic result and the number of times the first word appears in the multiple first texts, determine the importance of the first word; wherein the importance of the first word is positively correlated with the first logarithmic result and the number of times the first word appears in the multiple first texts.

[0020] In the above scheme, the conditional entropy determination module is also used to perform the following processing for each context word corresponding to the first word: determine the first word frequency of the context word based on the number of occurrences of the context word of the first word; determine the conditional entropy of the first word corresponding to the context word based on the first word frequency of each context word corresponding to the first word.

[0021] In the above scheme, the conditional entropy determination module is also used to perform logarithm processing on the first word frequency of each context word corresponding to the first word to obtain a second logarithmic result, multiply the second logarithmic result by the first word frequency to obtain a first multiplication result corresponding to the context word; and sum the opposite numbers of the first multiplication results corresponding to multiple context words to obtain the conditional entropy of the first word corresponding to the context word.

[0022] In the above scheme, the conditional entropy determination module is also used to sum the number of occurrences of multiple context words of the first words to obtain the total number of context words of the first words; and use the ratio of the number of occurrences of the context words of the first words to the total number of context words as the first word frequency of the context words.

[0023] In the above scheme, the word vector dimension determination module is also used to average the conditional entropies of multiple first words corresponding to the context words to obtain the average conditional entropy, and average the importance corresponding to multiple first words to obtain the average importance; based on the average conditional entropy, the average importance, the context window, and the number of first words, the word vector dimension is determined; wherein, the word vector dimension is positively correlated with the average conditional entropy, the average importance, the length of the context window, and the number of first words.

[0024] An embodiment of the present application provides an electronic device, comprising:

[0025] a memory for storing computer-executable instructions or computer programs;

[0026] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0027] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in the embodiment of the present application when executed by a processor.

[0028] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the data processing method provided in the embodiment of the present application is implemented.

[0029] The embodiments of the present application have the following beneficial effects:

[0030] Perform word segmentation on multiple first texts to obtain multiple different first words; for each first word, determine the importance of the first word based on the number of times the first word appears in multiple first texts. Each word has its own significance and contribution, so the importance of each word needs to be considered in dimensional planning. Words with higher importance require word vectors with larger dimensions to represent them, thereby improving the representation accuracy; for each first word, query the context words of the first word in the first text including the first word, wherein in the first text, the context words of the first word are words within the context window of the first word in the first text; for each first word, determine the conditional entropy of the first word corresponding to the context word based on the number of times the context word of the first word appears. Conditional entropy refers to the expected value of the uncertainty of the context word under the condition that the first word is known, which measures how much uncertainty of the context word has not been resolved under the premise that the first word is known. The lower the conditional entropy, the more concentrated the distribution of the context words of the first word is, and the lower the uncertainty is. On the contrary, the higher the conditional entropy, the more dispersed the distribution of the context words of the first word is, and the higher the uncertainty is. A higher dimension may be required to represent the complex semantics brought about by higher uncertainty. Therefore, the word vector needs to increase the dimension to accommodate more information. Based on the context window, the number of first words, the conditional entropy of each first word corresponding to the context word, and the importance of each first word, the word vector dimensions of multiple first texts are determined. From the above analysis, it can be seen that the importance of the first word and the conditional entropy of the first word corresponding to the context word are closely related to the word vector dimension. In this embodiment of the present application, not only the number of first words is taken into account, but also the importance of the first word, the conditional entropy of the first word corresponding to the context word, and the context window are considered. Therefore, the determined word vector dimension is adapted to the word distribution in multiple first texts, so that the first text can be more accurately represented later. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1This is a schematic diagram of the structure of the data processing system architecture provided by the embodiment of the present application;

[0032] Figure 2 is a structural diagram of a data processing device provided in an embodiment of the present application;

[0033] Figure 3A-3G Schematic diagram of the data processing method provided in the embodiment of the present application;

[0034] Figure 4 This is a schematic diagram of the process of determining word vector dimensions provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0036] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0038] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0039] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0040] The related data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0041] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0042] (1) Word vector dimension: a numerical parameter representing the semantic information contained in a single word vector in a word vector model, that is, how many numerical values constitute a word vector. It determines the complexity of the semantic features that a word vector can carry. Too low a dimension will lose information, and too high a dimension will be redundant. A suitable dimension can accurately express semantic associations.

[0043] (2) Word vector model: a technical tool for converting words in natural language into computer-processed numerical vectors. By analyzing massive text data, mining word semantic associations and context information, mapping words to a multi-dimensional vector space, and making semantically similar words close in space, it helps machines understand language.

[0044] (3) Corpus: an electronic resource library storing massive real language materials, covering various texts or voice data such as books and articles.

[0045] (4) Context word: a word within a certain range around the target word that has semantic association or grammatical connection with the target word. For example, in "apples are delicious", "apples" has no context before and "are delicious" after, which are context words, and they form a subject-predicate semantic association.

[0046] (5) Context window: a parameter in natural language processing that defines the range of associated words of a target word. It takes the target word as the center and defines a certain number of words before and after, such as 5 words before and after as the analysis area. It helps the word vector model accurately capture the semantic and grammatical connections between the target word and its neighboring words, and facilitates semantic understanding.

[0047] Word vector dimension refers to the number of numerical values contained in a vector when a word is mapped to a vector, which is an important parameter of the word vector model. High-dimensional word vectors can capture more rich semantic and grammatical features, for example, word vectors can reflect semantic similarity through the distance in high-dimensional space. However, too high a word vector dimension will increase the computational complexity and may lead to overfitting; and too low a dimension will not be able to fully represent the complexity of the data and may lose important information.

[0048] In the related technology, D≈logN is used for approximate estimation, where N represents the total number of words and D is an integer representing the word vector dimension. This scheme takes into account that the relationship between words grows exponentially with the vocabulary size, and the logarithmic relationship can better balance the expressive power and computational cost, that is, when N increases, logN grows relatively slowly, so that a certain expressive power can be maintained without excessively increasing the dimension. However, the related technology only considers the total number of words. If there are two sources of text collections, each text collection contains N words, but the N words in one text collection are the same, and the N words in the other text collection are different, according to the existing scheme, the same dimensional word vectors are used to represent the two text collections, which is obviously unreasonable. This single determination method ignores the internal word distribution characteristics of the text collection, and the determined word vector dimension brings a bottleneck to the accuracy of the word vector representation.

[0049] The embodiments of the present application provide a data processing method, apparatus, device, computer-readable storage medium, and computer program product. The word vector dimension determined by the present application is adapted to the word distribution in the first text, so that the first text can be more accurately represented. The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The device provided by the embodiment of the present application can be implemented as various types of terminals such as laptops, tablet computers, and desktop computers, and can also be implemented as a server. Below, an exemplary application when the device is implemented as a server will be described.

[0050] See also Figure 1 , Figure 1 This is a structural diagram of the data processing system architecture provided in an embodiment of the present application. In order to support a data processing application, in the data processing system 100, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The server 200 uploads the data to the database 500.

[0051] The server 200 obtains a plurality of first texts from the database 500 and performs word segmentation processing on the plurality of first texts to obtain a plurality of mutually different first words; for each first word, based on the number of occurrences of the first word in the plurality of first texts, determines the importance of the first word; for each first word, searches for a context word of the first word in the first text including the first word, wherein, in the first text, the context word of the first word is a word within a context window of the first word in the first text; based on the number of occurrences of the context word of the first word, determines the conditional entropy of the first word corresponding to the context word; based on the context window, the number of the first words, the conditional entropy of each first word corresponding to the context word, and the importance of each first word, determines the word vector dimensions of the plurality of first texts; the server 200 uploads the determined word vector dimensions to the database 500 and, based on the plurality of first texts, trains a word vector model corresponding to the word vector dimensions, wherein the word vector model is used to convert the first text into a word vector having the word vector dimensions. The server 200 deploys the trained word vector model in the terminal 400.

[0052] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0053] See also Figure 2 , Figure 2 2 is a structural diagram of a data processing device provided by an embodiment of the present application. The server 200 includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the server 200 are coupled together through a bus system 240. It can be understood that the bus system 240 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 240 .

[0054] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0056] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0057] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0058] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0059] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0060] A network communication module 252 for reaching other electronic devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0061] a presentation module 253 for enabling presentation of information (e.g., a user interface for operating the peripheral device and displaying content and information) via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with the user interface 230;

[0062] an input processing module 254 for detecting and translating one or more user inputs or interactions from one or more input devices 232.

[0063] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in software, Figure 2 A data processing apparatus 255 stored in the memory 250 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: a word segmentation module 2551, an importance determination module 2552, a context word determination module 2553, a conditional entropy determination module 2554, and a word vector dimension determination module 2555. These modules are logical, and thus can be combined or further split according to the implemented functions. The functions of each module will be described below.

[0064] In some other embodiments, the apparatus provided by the embodiments of the present application can be implemented in hardware. As an example, the apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to perform the data processing method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic elements.

[0065] In some embodiments, the terminal or server can implement the data processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP or an instant messaging APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0066] The following describes the data processing method provided by the embodiment of the present application. As mentioned above, the electronic device that implements the data processing method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0067] See also Figure 3A , Figure 3A This is a flow chart of the data processing method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained, Figure 3A The body of the step is the server.

[0068] In step 101, word segmentation is performed on a plurality of first texts to obtain a plurality of different first words.

[0069] In some embodiments, see Figure 3B , Figure 3A Step 101 shown can be implemented by the following steps 1011 to 1012, as shown below. Figure 3B Provide explanation.

[0070] In step 1011, word segmentation processing is performed on the plurality of first texts to obtain second words included in the plurality of first texts.

[0071] As an example, word segmentation processing first preprocesses the first text, removes special symbols, redundant spaces, labels and other noise data in the first text, and unifies the text encoding format to build a pure input environment for subsequent word segmentation. Then, an adapted word segmentation tool is selected, such as the Jieba word segmentation based on dictionary matching, or the BERT word segmentation model based on deep learning, and the word segmentation result of the first text is the second word. The first text here can be understood as the original input data in the natural language processing task, that is, the text set to be analyzed; the second word can be understood as the discrete semantic unit obtained after the word segmentation process, that is, the result of segmenting the first text.

[0072] In step 1012, duplicate removal is performed on the second words included in the plurality of first texts to obtain a plurality of different first words.

[0073] As an example, if there are repeated contents in the multiple second words obtained after word segmentation of multiple first texts, all the second words can be stored in the form of a set or hash table structure. The set and hash table structure have deduplication characteristics and can automatically filter out duplicate items; or the second words can be traversed and the words that have appeared can be recorded through dictionary key values. If the words are in the dictionary, they are skipped, and finally a set of non-repeating words is output, which is the first word.

[0074] Through the embodiments of the present application, word segmentation enables the machine to understand the text structure and semantic units in more detail. It removes duplicate words and extracts multiple different first words, which greatly simplifies the data size and effectively reduces information redundancy.

[0075] Continue to see Figure 3A In step 102, for each first word, the importance of the first word is determined based on the number of times the first word appears in multiple first texts.

[0076] In some embodiments, see Figure 3C , Figure 3A Step 102 shown can be implemented by the following steps 1021 to 1023, as shown below. Figure 3C Provide explanation.

[0077] In step 1021 , a first ratio of the total number of words in the plurality of first texts to the number of occurrences of the first word in the plurality of first texts is determined.

[0078] In step 1022, logarithmic processing is performed on the first ratio to obtain a first logarithmic result.

[0079] In step 1023 , the importance of the first word is determined based on the first logarithmic result and the number of times the first word appears in the plurality of first texts.

[0080] As an example, the importance of the first word is positively correlated with the first logarithmic result and the number of times the first word appears in the plurality of first texts. The calculation method of the importance is shown in the following formula (1).

[0081]

[0082] Among them, f(ω i ) represents word ω i That is, the total number of times the first word appears in the corpus, that is, the number of times the first word appears in multiple first texts, N is the total number of words in multiple first texts, that is, Figure 3BThe plurality of first texts are segmented in step 1011 to obtain a total number of a plurality of second words; is a first ratio, is a first logarithmic result; The importance of the word ω i is measured; the occurrence times of different words differ greatly, and high-frequency words and low-frequency words have different requirements for representing dimensions. High-frequency words often have more general language rules, and low-frequency words are more specific. Therefore, each word has its own significance and contribution, so the different weights of each word need to be considered when planning the dimensions. i reflects the importance level of the word ω i in the corpus, which helps to determine the required word vector dimension.

[0083] Through the embodiments of the present application, the significance and contribution of high-frequency words and low-frequency words are comprehensively considered, which avoids the excessive bias of high-frequency words caused by pure word frequency statistics and corrects the defect of directly ignoring low-frequency words, and can accurately measure the importance of the first word.

[0084] Continuing to refer to Figure 3A , in step 103, for each first word, the context words of the first word are queried in the first text including the first word.

[0085] As an example, in the first text, the context words of the first word are the words in the first text that belong to the context window of the first word, that is, the n words before and after the first word, and n is the context window. For example, for the sentence "I like to drink coffee", assuming that the first word is "drink", when the context window is 1, the context words of "drink" are only "like" and "coffee"; when the context window is 2, the context words of "drink" include "like", "coffee" and "I".

[0086] In step 104, for each first word, the conditional entropy of the first word corresponding to the context word is determined based on the occurrence times of the context words of the first word.

[0087] In some embodiments, referring to Figure 3D , Figure 3A The step 104 shown can be implemented by the following steps 1041 to 1042, which will be described below in combination with Figure 3D .

[0088] In step 1041, the first word frequency of the context word is determined based on the occurrence times of the context words of the first word.

[0089] As an example, step 1041 is performed for each context word corresponding to the first word.

[0090] In some embodiments, see Figure 3E , Figure 3D Step 1041 shown can be implemented by the following steps 1041A to 1041B, as shown below. Figure 3E Provide explanation.

[0091] In step 1041A, the number of occurrences of the context words of the multiple first words is summed up to obtain the total number of context words of the first word.

[0092] In step 1041B, the ratio of the number of occurrences of the context words of the first word to the total number of context words is used as the first word frequency of the context words.

[0093] As an example, the calculation method of the first word frequency of the context word is shown in formula (2).

[0094]

[0095] Among them, ω is the first word frequency of the context word, ω i,j The first word ω i The number of occurrences of the j-th context word, C i is the total number of context words of the first word.

[0096] Through the embodiments of the present application, the first word frequency of the context word of the first word can be accurately determined, and the first word frequency can objectively reflect the degree of association between the context word of the first word and the first word.

[0097] Continue to see Figure 3D In step 1042, based on the first word frequency of each context word corresponding to the first word, the conditional entropy of the context word corresponding to the first word is determined.

[0098] In some embodiments, see Figure 3F , Figure 3D Step 1042 shown can be implemented by the following steps 1042A to 1042C, as shown below. Figure 3F Provide explanation.

[0099] In step 1042A, the first word frequency of the context word is logarithmized to obtain a second logarithm result;

[0100] In step 1042B, the second logarithmic result is multiplied by the first word frequency to obtain a first multiplication result corresponding to the context word.

[0101] As an example, steps 1042A to 1042B are performed for each context word corresponding to the first word.

[0102] In step 1042C, the inverse numbers of the first multiplication results corresponding to the multiple context words are summed to obtain the conditional entropy of the context words corresponding to the first word.

[0103] As an example, the calculation method of conditional entropy is shown in the following formula (3).

[0104]

[0105] Among them, ω i,j The first word ω i The number of occurrences of the j-th context word, C i is the total number of context words of the first word; c i For the first word ω i The total number of context words after deduplication; Indicates the first word ω i The first word frequency of the j-th context word; is the second logarithmic result; is the first multiplication result, is the opposite of the first multiplication result; Indicates the first word ω i The conditional entropy h i .

[0106] As an example, the first word ω i The statistical value C corresponding to the context word i and c i , depends on the size of the context window X. Assume that the corpus, i.e. the first text, contains two texts, namely "I like to drink coffee." and "Xiao Ming likes to drink juice and milk very much." When X=1, the word "like" appears twice in the context of the word "drink", and "coffee" and "juice" appear once each. Therefore, C i =4,c i =3.

[0107] As an example, the first word ω i Conditional entropy It means that the first word is known. i The expected value of the uncertainty of the context words of the first word under the condition of ω. That is, it measures the uncertainty of the context words of the first word under the condition of ω. i Under the premise, how much uncertainty in the context words is not resolved. The lower the conditional entropy, the more likely the first word ω is. i The more concentrated the context words are, that is, the first word ω i The more likely it is to appear with a context word, the easier it is to predict. i The smaller the prediction difficulty value is. On the contrary, the higher the conditional entropy is, the higher the first word ω is. iThe more dispersed the context words are, the more likely the first word ω i The less likely it is to appear with a context word, the less likely it is to be predicted. i The greater the prediction difficulty value, the higher the dimension may be needed to represent this diversity, which means that the word vector needs to increase the dimension to accommodate more information. h is a holistic measure of the prediction difficulty of the word and helps determine the required word vector dimension.

[0108] Through the embodiments of the present application, the smaller the conditional entropy of the context word corresponding to the first word is, the more likely the context word is to appear at the same time as the first word, and the smaller the required word vector dimension is; the larger the conditional entropy of the context word corresponding to the first word is, the less likely the context word is to appear at the same time as the first word, and a higher word vector dimension is required to represent this diversity, that is, the word vector needs to increase its dimension to accommodate more information. Determining the accurate conditional entropy helps to determine the required word vector dimension.

[0109] By using the embodiments of the present application, the first word frequency of context words can be counted using objective data to quantify the degree of direct connection between each context word and the first word, avoiding subjective assignment bias while retaining the original information of the context. Determining the conditional entropy based on each first word frequency can comprehensively reflect the size of the required word vector dimension. Accurate conditional entropy helps determine the required word vector dimension.

[0110] Continue to see Figure 3A In step 105, the word vector dimensions of the plurality of first texts are determined based on the context window, the number of first words, the conditional entropy of each first word corresponding to the context word, and the importance of each first word.

[0111] In some embodiments, see Figure 3G , Figure 3A Step 105 shown can be implemented by steps 1051 to 1053 below. Figure 3G Provide explanation.

[0112] In step 1051, the conditional entropies of the context words corresponding to the multiple first words are averaged to obtain an average conditional entropy.

[0113] As an example, the calculation formula of the average conditional entropy is shown in the following formula (4).

[0114]

[0115] in, is the average conditional entropy, h i The first word ω i Corresponding to the conditional entropy of the context word, D is the number of times the first word appears in multiple first texts.

[0116] In step 1052, the importance corresponding to the plurality of first words is averaged to obtain an average importance.

[0117] As an example, the calculation formula for the average importance is shown in the following formula (5).

[0118]

[0119] in, is the average importance, γ i The first word ω i The corresponding importance, D, is the number of times the first word appears in the multiple first texts.

[0120] In step 1053 , the word vector dimension is determined based on the average conditional entropy, the average importance, the context window, and the number of first words.

[0121] As an example, the formula for determining the word vector dimension is shown in the following formula (6).

[0122]

[0123] Wherein, exp() represents the natural exponential function; X represents the size of the context window; D represents the number of occurrences of the first word in the plurality of first texts; is the average conditional entropy; is the average importance.

[0124] As an example, the word vector dimension is positively correlated with the average conditional entropy, the average importance, the length of the context window, and the number of the first words. The larger the context window X, the more diverse knowledge the model needs to learn, and the more difficult the model learning is. Therefore, a higher-dimensional vector is needed to represent the complex associations between words. The more times D the first word appears in multiple first texts, the more complex and diverse the words will be. Therefore, a higher-dimensional vector is also needed to reduce the learning difficulty of the model. However, the word vector dimension n is not linearly related to the context window X and the number of times the first word appears in multiple first texts. The initial dimension value of the word vector is obtained by multiplying the number of times the first word appears in multiple first texts D by the context window X and then taking the square root, that is, The initial dimension value is obtained from the representational features of the corpus, i.e. the first text. For a known corpus, i.e. the first text, the number of occurrences of the first word in multiple first texts, D, is constant, and X is set manually. On this basis, the average importance of the word is and the average conditional entropy of the word It also affects the required word vector dimension, the average conditional entropy of the word The larger the word is, the higher the word vector dimension is needed to represent this diversity, that is, the word vector needs to increase the dimension to accommodate more information; the average importance of the word The larger the value, the higher the dimension is needed to represent this importance. Therefore, using As The correction term is used to adjust the initial dimension value to obtain the word vector dimension n.

[0125] Through the embodiments of the present application, the conditional entropy and importance of the first word are averaged respectively, and the word vector dimension is obtained by combining the preset context window and the number of first words, thereby achieving adaptive adaptation of the word vector dimension and the semantic complexity of the text, significantly improving the accuracy of semantic representation.

[0126] exist Figure 3A After step 105 shown, a word vector model of corresponding word vector dimensions may be trained based on the plurality of first texts.

[0127] As an example, a word vector model is used to convert a first text into a word vector with a word vector dimension. When building a training model based on the word vector dimension, it is necessary to select an algorithm based on the predetermined word vector dimension, combined with the scale and semantic complexity of the first text, such as Word2Vec by capturing context distribution, or BERT using Transformer to dynamically model ambiguity; preprocess the word sequence after the first text is segmented, including building a vocabulary, mapping IDs, padding or truncating sequences, etc., and after inputting the word vector model, optimize the large-scale vocabulary calculation through negative sampling or hierarchical Softmax, and minimize the loss function through multiple rounds of iterations including adjusting the learning rate, batch size, etc. The loss function can be a cross-entropy function until the word vector model converges, and finally output a dense vector representation of each word in the specified dimension, and verify the semantic relevance between words through cosine similarity.

[0128] Through the embodiments of the present application, multiple first texts are segmented to obtain multiple different first words; for each first word, the importance of the first word is determined based on the number of times the first word appears in the multiple first texts. Each word has its own significance and contribution, so the importance of each word needs to be considered in dimensional planning. Words with higher importance require word vectors with larger dimensions for representation, thereby improving representation accuracy; for each first word, the context word of the first word is queried in the first text including the first word, wherein, in the first text, the context word of the first word is a word within the context window of the first word in the first text; for each first word, the conditional entropy of the first word corresponding to the context word is determined based on the number of times the context word of the first word appears. The conditional entropy refers to the expected value of the uncertainty of the context word under the condition that the first word is known, which measures how much uncertainty there is in the context word under the premise that the first word is known. The problem of uncertainty has not been solved. The lower the conditional entropy, the more concentrated the distribution of the context words of the first word is, and the lower the uncertainty is. On the contrary, the higher the conditional entropy, the more dispersed the distribution of the context words of the first word is, and the higher the uncertainty is. A higher dimension may be required to represent the complex semantics brought about by higher uncertainty. Therefore, the word vector needs to increase the dimension to accommodate more information. Based on the context window, the number of first words, the conditional entropy of each first word corresponding to the context word, and the importance of each first word, the word vector dimensions of multiple first texts are determined. From the above analysis, it can be seen that the importance of the first word and the conditional entropy of the first word corresponding to the context word are closely related to the word vector dimension. In this embodiment of the present application, not only the number of first words is taken into account, but also the importance of the first word, the conditional entropy of the first word corresponding to the context word, and the context window are considered. Therefore, the determined word vector dimension is adapted to the word distribution in multiple first texts, so that the first text can be more accurately represented later.

[0129] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0130] With the continuous development of natural language processing, word embedding models like Word2Vec and Glove have been gradually replaced by Transformer-based models like BERT. However, as a classic word embedding representation method, word embedding models continue to be effective in many scenarios, and there is still much room for improvement and reference. However, choosing the right word embedding dimension is a challenge. If the dimension is too large, the word embedding may be sparse and computationally complex. If the dimension is too small, it may not be sufficient to represent semantic meaning. Therefore, it is necessary to determine the appropriate word embedding dimension.

[0131] In practical applications, the dimensions of word vectors are generally 64, 100, 128, 256, 300, etc. For the same data, the effects will be slightly different when using different word vector dimensions. If you want to get the best word vector dimension for a batch of data, you need a lot of repeated training and testing to determine it. Obviously, this is not a cost-effective solution. In related technologies, one method is to use D≈logN approximate estimation, where N represents the total number of words, and D is an integer representing the dimension of the word vector. Considering that the relationship between words grows exponentially with the vocabulary size, the logarithmic relationship can better balance the expressive power and computational cost, that is, when N increases, logN grows slower, so that a certain expressive power can be maintained without excessively increasing the dimension; another solution is to use The advantage of approximate estimation is that it will not cause the dimension to grow too quickly, and it can also adapt to different changes in vocabulary size. In practical applications, the above two methods can only be used to preliminarily estimate the appropriate dimension range, and then adjust it based on the experimental results. Therefore, the defects of the above scheme are: first, it only considers the total number of words. For example, there are two sets of data, both containing N words. One set of data has the same N words, and the other set has different N words, but the calculated word vector dimensions are the same, which is obviously unreasonable; second, it can only give a preliminary estimate range.

[0132] The embodiment of the present application proposes a method for determining word vector dimensions, which can determine more appropriate word vector dimensions, reduce the number of experiments, and improve the accuracy of word vector representation. A plurality of factors that affect the dimensional value of the word vector are introduced, and a method for determining the dimensional value of the word vector is designed, which can more accurately determine the dimensional value of the word vector; for a corpus, that is, multiple first texts, different context windows are selected, and the embodiment of the present application can quickly and accurately give the word vector dimensional value required for training, thereby improving the accuracy of word vector representation.

[0133] See also Figure 4 , Figure 4 : This is a flow chart of determining the word vector dimension provided by an embodiment of the present application. Corpus, that is, the corpus for training word vector models such as Word2Vec and Glove, is assumed to contain a total of m texts. Total number of words, m texts in the corpus are segmented, and the total number of words obtained is assumed to be N. Among them, the word segmentation method can adopt jieba, LTP, etc. Number of unique words, the word segmentation results of the total number of words are deduplicated to obtain the number of unique words, which is assumed to be D. The calculation method of the word importance coefficient, that is, the importance γ, is shown in the following formula (7).

[0134]

[0135] Among them, f(ω i ) represents word ω iThe total number of occurrences in the corpus; What is measured is the word ω i γ measures the average importance of all unique words, i.e., the importance coefficient of a corpus word. Different words vary greatly in their frequency of occurrence, and high-frequency and low-frequency words require different representational dimensions. High-frequency words tend to exhibit more general linguistic patterns, while low-frequency words are more specific. Therefore, each word has its own significance and contribution, so the different weights of each word need to be considered when planning dimensions. γ reflects the importance of words in the corpus and helps determine the required representational dimensions.

[0136] Conditional entropy of words The calculation method of is shown in the following formula (8).

[0137]

[0138] Among them, C i Representation word ω i The context words have a total of C i c i : represents word ω i There are c unique words in the context words i , c i It is C i The number of words after removing duplicate context words; ω i,j Representation word ω i The number of j-th context words, such as "I like to drink coffee", assuming ω i When the word “drink” is “drink” and the size of the context window X is 1, the context words of “drink” only include “like” and “coffee”; when the size of the context window X is 2, the context words of “drink” include “like”, “coffee” and “I”. Representation word ω i The frequency of the j-th context word; Representation word ω i The conditional entropy of represents the average conditional entropy of all unique words

[0139] Word ω i Conditional entropy It means that in the known word ω i The expected value of the uncertainty of the context word under the condition of ω. That is, it measures the uncertainty of the context word under the condition of ω. i Under the premise, how much uncertainty in the context word is still unresolved. The lower the conditional entropy, the more likely the word ω is to be i The more concentrated the context words are, the more concentrated the word ω is. i The more it tends to appear with a context word, the easier it is to predict. iThe smaller the prediction difficulty value is. On the contrary, the higher the conditional entropy is, the i The more dispersed the context words are, the more likely the word ω i The less likely it is to appear with a context word, the less likely it is to be predicted. i The greater the prediction difficulty value, the higher the dimension may be needed to represent this diversity, that is, the word vector needs to increase the dimension to accommodate more information.

[0140] The method for determining the dimension value n of the word vector is shown in the following formula (9).

[0141]

[0142] Where exp() represents the natural exponential function; X represents the size of the context window. i The statistical value C corresponding to the context word i and c i , depends on the size of the context window X. Assume that the corpus contains two texts, namely "I like to drink coffee." and "Xiao Ming likes to drink juice and milk very much." When X=1, the context words of the word "drink": "like" appears twice, "coffee" and "juice" appear once each, so C i =4,c i =3.

[0143] The larger the context window X, the more diverse the knowledge the model needs to learn, and the more difficult it is to learn. Therefore, a higher-dimensional vector is needed to represent the complex associations between words. The larger the number of unique words D, the greater the complexity and diversity between words. Therefore, a higher-dimensional vector is also needed to reduce the difficulty of learning the model. However, the dimension value n of the word vector is not linearly related to the context window X and the number of unique words D. The initial dimension value of the word vector is obtained by multiplying the number of unique words D by the context window X and then taking the square root, that is, The initial dimension value is obtained from the representational characteristics of the corpus. For a given corpus, the number of unique words D is constant, and X is set manually. On this basis, the importance coefficient γ of the word and the conditional entropy of the word It also affects the required representation dimension, the conditional entropy The larger the value is, the higher the dimension is needed to represent this diversity, that is, the word vector needs to increase the dimension to accommodate more information; the larger the value of the word importance coefficient γ is, the higher the dimension is needed to represent this importance. Compared with the representational feature context window X and the number of unique words D, the two are closer to the essence. Therefore, using As The correction term is used to adjust the initial dimension value to obtain the dimension value n of the word vector.

[0144] Model training: For a corpus, after determining the size of the context window and the dimension of the word vector, a word vector model can be trained using methods such as Word2Vec and Glove.

[0145] The embodiment of the present application first introduces multiple factors that affect the dimensional value of the word vector, and designs a method for determining the dimensional value of the word vector, which can more accurately determine the dimensional value of the word vector; second, for a corpus, different context windows are selected, and the embodiment of the present application can quickly and accurately give the word vector dimensional value required for training, thereby improving the accuracy of the word vector representation.

[0146] The embodiments of the present application can be used for text retrieval. The word vector represents the semantics of the word and can be used for text classification, clustering, and word similarity calculation. In intelligent quality inspection scenarios, agents often use some non-compliant words, which lead to customers asking questions, such as "false information" scenarios and "unprofessional language" scenarios. In order to improve the service level of the agents, it is necessary to detect the agents' words. Therefore, it is necessary to train semantic models for these scenarios separately, and the training model requires training data. In order to quickly and accurately salvage a batch of relevant data for annotation, a text retrieval function is required. Specifically, first, a batch of agent conversations in this scenario are salvaged as a corpus, and the total number of words is obtained by segmenting the text in the corpus. The segmentation results are deduplicated to obtain the number of unique words in the corpus; then, the number of times each unique word appears in the corpus is counted, and the importance of each unique word is calculated by the method in the article, and the average importance of all unique words can be obtained; then, for each unique word, the total number of its context words and the number of unique words therein are counted, and the conditional entropy of each unique word is calculated by the method in the article, and the average conditional entropy of all unique words is obtained; finally, according to the method for determining the word vector dimension value proposed in the embodiment of the present application, the word vector dimension value of the corpus under a certain value of the context window is obtained, and the size of the context window and the dimension of the word vector are determined. Then, a word vector model can be trained using methods such as Word2Vec and Glove. We can select the agent's speech in the past month, remove duplicates, and use the word vector model to perform text representation on each agent's speech to obtain the agent's speech vector library. Then, we can characterize the non-compliant speech in a certain scenario and calculate the similarity with the vectors in the agent's speech vector library. We can select the most similar batch of texts, perform data annotation, and finally train the semantic model for the scenario to detect whether the agent's speech involves non-compliant expressions in the scenario.

[0147] The following continues to describe the exemplary structure of the data processing device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, see Figure 2, the software modules stored in the data processing device 255 of the memory 250 may include:

[0148] A word segmentation module 2551 is used to perform word segmentation processing on the multiple first texts to obtain multiple different first words;

[0149] an importance determination module 2552 for determining, for each first word, the importance of the first word based on the number of occurrences of the first word in the plurality of first texts;

[0150] a context word determination module 2553 configured to, for each first word, search a first text containing the first word for a context word of the first word, wherein in the first text, the context word of the first word is a word within a context window of the first word in the first text;

[0151] a conditional entropy determination module 2554 configured to determine, for each first word, a conditional entropy of the first word corresponding to the context word based on the number of occurrences of the context word of the first word;

[0152] The word vector dimension determination module 2555 is used to determine the word vector dimension of the multiple first texts based on the context window, the number of the first words, the conditional entropy of each first word corresponding to the context word, and the importance of each first word.

[0153] In some embodiments, the word segmentation module 2551 is further used to perform word segmentation processing on the multiple first texts to obtain second words included in the multiple first texts; and to perform deduplication processing on the second words included in the multiple first texts to obtain multiple different first words.

[0154] In some embodiments, the importance determination module 2552 is also used to determine a first ratio of the total number of words in the multiple first texts to the number of times the first word appears in the multiple first texts; perform logarithmic processing on the first ratio to obtain a first logarithmic result; determine the importance of the first word based on the first logarithmic result and the number of times the first word appears in the multiple first texts; wherein the importance of the first word is positively correlated with the first logarithmic result and the number of times the first word appears in the multiple first texts.

[0155] In some embodiments, the conditional entropy determination module 2554 is further used to perform the following processing for each context word corresponding to the first word: determining the first word frequency of the context word based on the number of occurrences of the context word of the first word; and determining the conditional entropy of the first word corresponding to the context word based on the first word frequency of each context word corresponding to the first word.

[0156] In some embodiments, the conditional entropy determination module 2554 is further used to perform logarithmic processing on the first word frequency of each context word corresponding to the first word to obtain a second logarithmic result, multiply the second logarithmic result by the first word frequency to obtain a first multiplication result corresponding to the context word; and sum the opposite numbers of the first multiplication results corresponding to multiple context words to obtain the conditional entropy of the first word corresponding to the context word.

[0157] In some embodiments, the conditional entropy determination module 2554 is further used to sum the number of occurrences of multiple context words of the first word to obtain the total number of context words of the first word; and use the ratio of the number of occurrences of the context words of the first word to the total number of context words as the first word frequency of the context word.

[0158] In some embodiments, the word vector dimension determination module 2555 is also used to average the conditional entropies of multiple first words corresponding to the context words to obtain an average conditional entropy, and average the importance corresponding to multiple first words to obtain an average importance; based on the average conditional entropy, the average importance, the context window, and the number of first words, determine the word vector dimension; wherein the word vector dimension is positively correlated with the average conditional entropy, the average importance, the length of the context window, and the number of first words.

[0159] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in the embodiment of the present application.

[0160] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data processing method provided by the embodiment of the present application.

[0161] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0162] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0163] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0164] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0165] In summary, through the embodiment of the present application, multiple first texts are segmented to obtain multiple different first words, which greatly simplifies the data scale and effectively reduces information redundancy; for each first word, the importance of the first word is determined based on the number of times the first word appears in multiple first texts. Each word has its own significance and contribution, so the importance of each word needs to be considered in dimensional planning. Words with higher importance require word vectors with larger dimensions for representation, thereby improving representation accuracy; for each first word, the context word of the first word is queried in the first text including the first word, wherein, in the first text, the context word of the first word is the word within the context window of the first word in the first text; for each first word, based on the number of occurrences of the context word of the first word, the conditional entropy of the first word corresponding to the context word is determined. The conditional entropy refers to the expected value of the uncertainty of the context word under the condition that the first word is known, which measures the uncertainty under the premise that the first word is known. , how much uncertainty in the context words has not been resolved, the lower the conditional entropy, the more concentrated the distribution of the context words of the first word, the lower the uncertainty, on the contrary, the higher the conditional entropy, the more dispersed the distribution of the context words of the first word, the higher the uncertainty, it may require a higher dimension to represent the complex semantics brought about by higher uncertainty, so the word vector needs to increase the dimension to accommodate more information; based on the context window, the number of first words, the conditional entropy of each first word corresponding to the context word, and the importance of each first word, the word vector dimensions of multiple first texts are determined. From the above analysis, it can be seen that the importance of the first word and the conditional entropy of the first word corresponding to the context word are closely related to the word vector dimension. In this embodiment of the present application, not only the number of first words is taken into account, but also the importance of the first word, the conditional entropy of the first word corresponding to the context word and the context window are considered. Therefore, the determined word vector dimension is adapted to the word distribution in multiple first texts, so that the first text can be more accurately represented later.

[0166] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Performing word segmentation processing on the plurality of first texts to obtain a plurality of mutually different first words; For each of the first words, determining the importance of the first word based on the number of occurrences of the first word in the plurality of first texts; For each of the first words, searching a first text including the first word for a context word of the first word, wherein in the first text, the context word of the first word is a word within a context window of the first word in the first text; For each of the first words, determining a conditional entropy of the first word corresponding to the context word based on the number of occurrences of the context word of the first word; The word vector dimensions of the plurality of first texts are determined based on the context window, the number of the first words, the conditional entropy of each of the first words corresponding to the context words, and the importance of each of the first words.

2. The method according to claim 1, characterized in that The determining the importance of the first word based on the number of occurrences of the first word in the plurality of first texts includes: determining a first ratio of a total number of words in the plurality of first texts to a number of occurrences of the first word in the plurality of first texts; performing logarithmic processing on the first ratio to obtain a first logarithmic result; determining the importance of the first word based on the first logarithmic result and the number of occurrences of the first word in the plurality of first texts; The importance of the first word is positively correlated with the first logarithmic result and the number of times the first word appears in the multiple first texts.

3. The method according to claim 1, characterized in that The determining, based on the number of occurrences of the context word of the first word, the conditional entropy of the first word corresponding to the context word includes: Performing the following processing on each context word corresponding to the first word: determining a first word frequency of the context word based on the number of occurrences of the context word of the first word; Based on a first word frequency of each of the context words corresponding to the first word, a conditional entropy of the first word corresponding to the context words is determined.

4. The method according to claim 3, characterized in that The determining, based on the first word frequency of each of the context words corresponding to the first word, the conditional entropy of the first word corresponding to the context word includes: For each context word corresponding to the first term, performing logarithm processing on a first word frequency of the context word to obtain a second logarithm result, and multiplying the second logarithm result by the first word frequency to obtain a first multiplication result corresponding to the context word; The inverse numbers of the first multiplication results corresponding to the plurality of context words are summed to obtain the conditional entropy of the first word corresponding to the context word.

5. The method according to claim 3, characterized in that The determining, based on the number of occurrences of the context word of the first word, a first word frequency of the context word includes: summing the number of occurrences of a plurality of context words of the first word to obtain a total number of context words of the first word; The ratio of the number of occurrences of the context words of the first word to the total number of the context words is used as the first word frequency of the context words.

6. The method according to claim 1, characterized in that The determining of the word vector dimension based on the context window, the number of the first words, the conditional entropy of each of the first words corresponding to the context words, and the importance of each of the first words includes: averaging the conditional entropies of the first words corresponding to the context words to obtain an average conditional entropy, and averaging the importances corresponding to the first words to obtain an average importance; Determining the word vector dimension based on the average conditional entropy, the average importance, the context window, and the number of the first words; The word vector dimension is positively correlated with the average conditional entropy, the average importance, the length of the context window, and the number of the first words.

7. A data processing device, characterized in that: The device comprises: A word segmentation module, configured to perform word segmentation processing on the plurality of first texts to obtain a plurality of different first words; an importance determination module, configured to determine, for each of the first words, the importance of the first word based on the number of occurrences of the first word in the plurality of first texts; a context word determination module, configured to, for each first word, search a first text containing the first word for a context word of the first word, wherein in the first text, the context word of the first word is a word within a context window of the first word in the first text; a conditional entropy determination module, configured to determine, for each first word, a conditional entropy of the first word corresponding to the context word based on the number of occurrences of the context word of the first word; A word vector dimension determination module is used to determine the word vector dimension of the multiple first texts based on the context window, the number of the first words, the conditional entropy of each first word corresponding to the context word, and the importance of each first word.

8. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the method according to any one of claims 1 to 6 when executing computer-executable instructions or computer programs stored in the memory.

9. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images