A feature fusion method and device in language recognition and a computer storage medium

Abstract

This application provides a feature fusion method, apparatus, and computer storage medium for language recognition. According to the feature fusion scheme provided in this application, the text to be recognized is obtained, the character order and text length of the characters contained in the text to be recognized are determined, and character temporal features, coding block features, word features, and character features of the text to be recognized are generated respectively. The order of the feature units contained in the character temporal features, coding block features, word features, and character features is the same as the character order, and the length of the character temporal features, coding block features, word features, and character features is the same as the text length. The character temporal features, coding block features, word features, and character features are fused to generate a multi-feature representation vector, realizing feature extraction and fusion of the text to be recognized from four dimensions: characters, words, coding blocks, and character temporal sequences.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a feature fusion method, apparatus, and computer storage medium for language recognition. Background Technology

[0002] In language recognition, self-attention-based neural network models are commonly used to extract character or word features from the text as input. However, this approach relies on a single feature set and often neglects local features, resulting in poor model performance.

[0003] For example, high-frequency characters in character features lack discriminative power, making it difficult to distinguish between languages ​​that share high-frequency characters; at the same time, the vocabulary of word features is too large to cover all words in all languages, resulting in low prediction accuracy for texts containing low-frequency words in practical applications; and in language recognition, training corpora often use long texts, which often ignore the local features contained in the corpus, resulting in poor recognition performance of the trained model for short texts.

[0004] Therefore, a more comprehensive feature fusion scheme that integrates multiple features is needed to improve the accuracy of language recognition. Summary of the Invention

[0005] In view of this, embodiments of this application provide a more comprehensive feature fusion scheme that integrates multiple features to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of the embodiments of this application, a feature fusion method for language recognition is provided, comprising: acquiring a text to be recognized; determining the character order and text length of the characters contained in the text to be recognized; generating character temporal features, coding block features, word features, and character features of the text to be recognized, wherein the order of feature units contained in the character temporal features, coding block features, word features, and character features is the same as the character order, and the length of the character temporal features, coding block features, word features, and character features is the same as the text length; and fusing the character temporal features, coding block features, word features, and character features to generate a multi-feature representation vector.

[0007] According to a second aspect of the embodiments of this application, a feature fusion apparatus for language recognition is provided, comprising: a determining module for acquiring text to be recognized and determining the character order and text length of the characters contained in the text to be recognized; a feature generating module for generating character temporal features, coding block features, word features, and character features of the text to be recognized, wherein the order of feature units contained in the character temporal features, coding block features, word features, and character features is the same as the character order, and the length of the character temporal features, coding block features, word features, and character features is the same as the text length; and a fusion module for fusing the character temporal features, coding block features, word features, and character features to generate a multi-feature representation vector.

[0008] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the feature fusion method described in the first aspect.

[0009] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the feature fusion method as described in the first aspect.

[0010] According to the feature fusion scheme provided in this application embodiment, the text to be identified is obtained, the character order and text length contained in the text to be identified are determined, and character temporal features, coding block features, word features and character features of the text to be identified are generated respectively. The order of the feature units contained in the character temporal features, coding block features, word features and character features is the same as the character order, and the length of the character temporal features, coding block features, word features and character features is the same as the text length. The character temporal features, coding block features, word features and character features are fused to generate a multi-feature representation vector, realizing feature extraction and fusion of the text to be identified from four dimensions: characters, words, coding blocks, and character temporal sequences. This allows the extracted multi-feature representation vector to integrate local and global features. Training and recognition of the model based on the fused multi-feature representation vector can significantly increase the accuracy of language recognition. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0012] Figure 1 A flowchart illustrating a feature fusion method for language recognition provided in an embodiment of this application;

[0013] Figure 2 This is a schematic diagram illustrating the generated features provided in an embodiment of this application;

[0014] Figure 3 This is a schematic diagram illustrating model training based on multiple feature representation vectors, provided as an embodiment of this application.

[0015] Figure 4 This is a schematic diagram illustrating multi-head self-attention model training based on multiple feature representation vectors, as provided in an embodiment of this application.

[0016] Figure 5a A schematic diagram of a window involving a character provided in an embodiment of this application;

[0017] Figure 5b A schematic diagram of a window related to a word provided in an embodiment of this application;

[0018] Figure 6 A schematic diagram of a feature fusion device for language recognition provided in an embodiment of this application;

[0019] Figure 7 This is a schematic diagram of the structure of an electronic device according to Embodiment 5 of this application. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0021] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings. For example... Figure 1 As shown, Figure 1 A flowchart illustrating a feature fusion method for language recognition provided in this application embodiment includes:

[0022] S101, Obtain the text to be recognized, and determine the character order and text length of the characters contained in the text to be recognized.

[0023] Character order refers to the order of characters in the text to be recognized. The characters in the text to be recognized also include the separators between characters, such as spaces, the end-of-text marker, etc.

[0024] For example, the text "Clé USB" could contain the characters "C", "l", "é", "[B]", "U", "S", "B", and "[E]", where "[B]" is the separator and "[E]" is the end-of-line character. The order of the characters is from "C" to "[E]".

[0025] The text length is the number of characters from the character "C" to the character "[E]". For the text "Clé USB", the text length is 8.

[0026] S103, generate the character temporal features, encoding interval features, word features and character features of the text to be identified respectively.

[0027] The length of character sequence features, coded block features, word features, and character features being the same as the length of the text means that each feature contains multiple feature units of the same length as the text (for example, for the text "CléUSB" with a length of 8, the generated features contain 8 feature units). The feature units contained in the same feature correspond one-to-one with the characters contained in the text, and the order of the feature units also corresponds to the order of the characters. See also Figure 2 , Figure 2 This is a schematic diagram of the generated features provided in the embodiments of this application.

[0028] Specifically, character features refer to the individual characters contained in the text to be identified. This includes delimiters and terminators. For example, for the text "Clé USB", the corresponding character features can be obtained based on the previously determined character order.

[0029] Character temporal features characterize the order of characters in the text to be recognized. For example, for the text "Clé USB", the sequential character temporal features are encoded from "0" to "7", meaning the generated character temporal feature length is 8, the same as the text length. See details. Figure 2 The diagram illustrates the temporal characteristics.

[0030] The encoding range characteristic refers to the encoding range corresponding to the character when it is encoded in Unicode. For example, the character "é" uses the Latin Supplement (LS) encoding range, therefore, it can be encoded using... <ls>The character "[B]" is the separator, and its corresponding encoding can also be [B]. The encoding of the terminating character [E] can also be [E]. Meanwhile, the other characters in "Clé USB" all use the Basic Latin encoding range, meaning their encoding is... <bl>See details. Figure 2 The diagram illustrates the coding interval features.

[0031] Similarly, other language encodings, including Chinese, Arabic, Japanese, and Korean, actually have different encoding ranges in Unicode. Therefore, depending on the language of the characters, there will be corresponding encoding range characteristics. In other words, the encoding range characteristics are strongly correlated with the language of the characters, and the obtained encoding range characteristics are the same as the length of the text.

[0032] The length of the word features is also the same as the length of the text. The text in the text to be identified can be divided according to delimiters; characters between two delimiters belong to the same word. Then, the feature units belonging to the same word can be encoded accordingly.

[0033] In one implementation, feature units belonging to the same word can be encoded the same way. For example, for the feature units in "Clé USB", which contain the words "Clé" and "USB", the characters "C", "l", and "é" can have the same encoding "Clé", such as... Figure 2 As shown in the image.

[0034] In another implementation, the word can be further segmented based on its constituent words (including prefixes, suffixes, and roots). For example, the word "review" can be segmented as "re" (prefix) + "view," thus assigning the word encoding for the feature units corresponding to the characters r and e as "re," and assigning the word encoding for the feature units corresponding to the characters v, i, e, and w as "view." Since prefixes, suffixes, and roots are often high-frequency components of a word, this approach can cover more words within the same vocabulary size, while also increasing the model's robustness to input features.

[0035] In addition, it should be noted that, for Figure 2 The character temporal features, encoding interval features, word features, and character features obtained in the dataset are illustrated with schematic encoding diagrams for each feature unit. In practice, the encoding of each feature unit is a vector with a specified dimension, and the specified dimension of the encoding used for the corresponding feature unit is the same in each feature. For example, for the first character "C", its corresponding encodings in each feature unit are "C", "0", "Clé", and "". <bl>However, in reality, the encoding corresponding to each feature unit uses a 128-dimensional vector, namely "C", "0", "Clé", and "...". <bl>"All are represented by 128-dimensional vectors."

[0036] S105, the character temporal features, coded block features, word features and character features are fused to generate a multi-feature representation vector.

[0037] The specific fusion method involves generating multi-feature representation vectors from the character temporal features, coded block features, word features, and character features, and then summing them to obtain a multi-feature representation vector. Clearly, in this method, the length of the resulting multi-feature representation vector remains the same as the text length.

[0038] In one implementation, the character temporal features, coded block features, word features, and character features can be directly concatenated to generate a multi-feature representation vector, and the length of the resulting multi-feature representation vector is the text length * 4.

[0039] In addition, in one approach, the obtained multi-feature representation vector can be subjected to a linear or nonlinear transformation to obtain the corresponding transformed multi-feature representation vector.

[0040] For example, for a multi-feature representation vector of size 1*d (where d is the text length), it can be transformed as follows: Transformed multi-feature representation vector = Multi-feature representation vector * Transformation function + Offset matrix, where the transformation function is a matrix of size d*L, and the offset matrix is ​​of size 1*L, resulting in a transformed multi-feature representation vector of size 1*L, thus achieving a linear transformation. Alternatively, the transformation function can be a trigonometric function, and the offset matrix can be a matrix of size 1*d, in which case a non-linear transformation can be achieved. The specific linear or non-linear transformation method can be selected based on actual needs, choosing the transformation function and offset matrix to achieve better feature extraction, which is beneficial for subsequent model training and recognition.

[0041] According to the feature fusion scheme provided in this application embodiment, the text to be identified is obtained, the character order and text length of the characters contained in the text to be identified are determined, and character temporal features, coding block features, word features, and character features of the text to be identified are generated respectively. The order of the feature units contained in the character temporal features, coding block features, word features, and character features is the same as the character order, and the length of the character temporal features, coding block features, word features, and character features is the same as the text length. The character temporal features, coding block features, word features, and character features are fused to generate a multi-feature representation vector, realizing feature extraction and fusion of the text to be identified from four dimensions: characters, words, coding blocks, and character temporal sequences. This allows the extracted multi-feature representation vector to integrate local and global features. Training and recognition of the model based on the fused multi-feature representation vector can significantly increase the accuracy of language recognition.

[0042] Furthermore, subsequent model training can be performed based on the obtained multi-feature representation vectors, such as... Figure 3 As shown, Figure 3 A schematic diagram illustrating model training based on multiple feature representation vectors, provided in this application embodiment, includes:

[0043] S301, determine the transpose vector of the multi-feature representation vector, multiply the transpose vector of the multi-feature representation vector with the multi-feature representation vector to generate the original information matrix.

[0044] Specifically, the transpose vector of the multi-feature representation vector is first determined (the length of the transpose vector is d*1), and then the transpose vector is multiplied by the multi-feature representation vector to obtain the original information matrix of size d*d.

[0045] It's easy to understand that since the multi-feature representation vector machine transpose is obtained by adding up the local information, the diagonal portion of the resulting original information matrix contains the local information corresponding to each character (the diagonal part takes the square of each coding unit in the multi-feature representation vector). Specifically, this includes the local information of the character's temporal features, coding interval features, word features, and character features. The other parts contain the global information generated when the information of each character interacts.

[0046] S303, determine multiple masking matrices of the same size as the original information matrix, wherein the multiple masking matrices contain matrix window masks of different sizes.

[0047] Therefore, multiple masking matrices of the same size as the original information matrix (the number of masking matrices is related to the number of multi-head attention models used in the model) can be used to extract local and global information respectively. Each masking matrix contains a matrix window mask, and these matrix window masks can have multiple different sizes. Within the same masking matrix, the matrix window masks extend diagonally towards the lower right corner. The matrix window mask is used to assign values ​​to relevant elements within the masking matrix; elements within the matrix window mask have a value of 0, and elements outside the matrix window mask have a value of negative infinity.

[0048] S305, add the original information matrix and the multiple masking matrices respectively to obtain multiple feature matrices.

[0049] Then, the original information matrix and the multiple masking matrices are added together to obtain multiple feature matrices. For example... Figure 3 As shown, Figure 3 This diagram illustrates multiple different masking matrices provided in an embodiment of this application. In the diagram, the matrix window masks in each masking matrix are 1*1, 2*2, 3*3, and d*d, respectively.

[0050] Since the values ​​inside the masking window of the masking matrix are 0 and the values ​​outside the masking window are negative infinity, after addition, the values ​​inside the masking window of the resulting feature matrix are still the values ​​corresponding to the original information matrix, while the values ​​outside the masking window are negative infinity.

[0051] S309, normalize the multiple feature matrices respectively to obtain multiple normalized feature matrices, and train the model based on the multiple normalized feature matrices.

[0052] When normalizing the eigenvalue matrix, negative infinity is normalized to 0, while for other values, the sum of the normalized values ​​is 0. For example, for the eigenvalue matrix... After normalization, its value becomes This allows for subsequent model training based on multiple normalized feature matrices.

[0053] Specifically, the normalized feature matrix obtained is still of size d*d. At this time, for each normalized feature matrix, it is multiplied with the transpose of the multi-feature representation vector (of size d*1) to generate multiple corresponding output vectors (of size d*1). Thus, multiple output vectors can be generated, and then the multiple output vectors can be merged to generate an output matrix (of size d*n, where n is the number of multi-heads used in the model). Then, the text to be identified can be identified based on the output matrix.

[0054] like Figure 4 As shown, Figure 4 This diagram illustrates a multi-head self-attention model training method based on multi-feature representation vectors, as provided in an embodiment of this application. In this diagram, Q, K, and V all represent input multi-feature representation vectors with a size of 1*d, where d is the text length. After performing a corresponding linear transformation (the linear transformations used may be the same or different) and then transposing them, multiple transposed vectors Q can be obtained. h K h and V h And so on, and then the corresponding original information matrix can be obtained based on the transpose vector, and then multiple feature matrices can be obtained by adding the mask matrix containing matrix window masks of different scales.

[0055] It should be noted that Q, K, and V all represent the same input (i.e., a multi-feature representation vector), but Q... h K h and V h These are the transpose matrices of Q, K, and V after linear transformations, respectively. However, since the transformations used in each linear transformation are different, Q... h K h and V h In reality, they can be the same or different.

[0056] In this schematic diagram, matrix window masks of sizes 1*1, 2*2, and 3*3, as well as multiple d*d matrix window masks (not shown in the diagram), are added to the original information matrix to obtain multiple different feature matrices. After normalizing the multiple feature matrices, they are multiplied by their transpose vectors to obtain multiple output vectors. These multiple output vectors are then combined to generate an output matrix, which is used to determine the language of the text to be identified.

[0057] In one implementation, for multiple masking matrices containing matrix window masks of different sizes, the sizes can be increased sequentially from 1*1 to N*N, where N is less than the text length d. The size of the matrix window masks in the remaining masking matrices is equal to the size of the original information matrix. This allows for the extraction of local features at different scales while preserving global features.

[0058] For example, when N = d / 2, i.e., when d is 8, we can take four matrix window masks of sizes from 1*1 to 4*4, as well as four 8*8 matrix window masks (in this case, the mask matrix = 0), thus obtaining local features at different scales and achieving better fusion of local features, such as... Figure 5a As shown, Figure 5a This is a schematic diagram of a window involving characters provided in an embodiment of this application. Due to the obstruction of separators and end characters, in Figure 5a In the example, for the character "U", the feature obtained by using a 3*3 masking window is actually a fusion of the local features between the two separators that are covered by the masking window.

[0059] In one implementation, since the text used for model training contains words of varying lengths, word-granularity masking can be used to enhance the model's feature fusion at the word level. Specifically, the word lengths in the text to be identified are first determined, and a matrix window mask matching the word lengths is then determined within the masking matrix.

[0060] For example, given the text "Clé USB", if we determine that the lengths of the words "Clé" and "USB" are both 3, then we can use four windows of different sizes along the diagonal: 3x3, 1x1 (corresponding to the separator), 3x3, and 1x1 (corresponding to the ending character). Each window in the matrix window mask will then correspond to one word. Figure 5b As shown, Figure 5b This diagram illustrates a window related to a word as provided in an embodiment of this application. In this diagram, four windows, corresponding to the word length, are sequentially arranged along the diagonal of the masking matrix. This method allows for flexible window division based on the word length in the input text, achieving effective fusion of local features at the word level.

[0061] The feature fusion method of this embodiment can be executed by any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs.

[0062] In a second aspect of this application, a feature fusion apparatus for language recognition is also provided, such as... Figure 6 As shown, Figure 6 A schematic diagram of a feature fusion device for language recognition provided in this application embodiment includes:

[0063] The determination module 601 acquires the text to be identified and determines the character order and text length of the characters contained in the text to be identified;

[0064] The feature generation module 603 generates character temporal features, coding block features, word features, and character features of the text to be identified, respectively. The order of the feature units contained in the character temporal features, coding block features, word features, and character features is the same as the character order, and the length of the character temporal features, coding block features, word features, and character features is the same as the length of the text.

[0065] The fusion module 605 fuses the character temporal features, coded block features, word features, and character features to generate a multi-feature representation vector.

[0066] Optionally, the fusion module 605 adds the character temporal features, coding block features, word features, and character features to generate a multi-feature representation vector; or, it concatenates the character temporal features, coding block features, word features, and character features to generate a multi-feature representation vector.

[0067] Optionally, the fusion module 605 performs a linear or nonlinear transformation on the generated multi-feature representation vector to obtain the transformed multi-feature representation vector.

[0068] Optionally, the feature generation module 603 determines the sub-words contained in the words of the text to be identified and the order of the sub-words, wherein the sub-words include prefixes, suffixes or roots contained in the words; segments the words according to the characters and generates word features of the text to be identified according to the order of the sub-words.

[0069] Optionally, the device further includes a model training module 607, which determines the transpose vector of the multi-feature representation vector, multiplies the transpose vector of the multi-feature representation vector with the multi-feature representation vector to generate an original information matrix; determines multiple masking matrices of the same size as the original information matrix, wherein the multiple masking matrices contain matrix window masks of different sizes; adds the original information matrix and the multiple masking matrices respectively to obtain multiple feature matrices; normalizes the multiple feature matrices respectively to obtain multiple normalized feature matrices, and performs model training based on the multiple normalized feature matrices.

[0070] Optionally, in the model training module 607, the size of the matrix window mask in the N masking matrices increases sequentially from 1*1 to N*N, where N is a positive integer less than the text length; the size of the matrix window mask in the remaining masking matrices is equal to the size of the original information matrix.

[0071] Optionally, in the model training module 607, the word length of the words contained in the text to be recognized is determined; and a matrix window mask matching the word length is determined in the masking matrix.

[0072] Optionally, the model training module 607 multiplies the multiple normalized feature matrices with the transpose of the multiple feature representation vectors to generate multiple corresponding output vectors; and merges the multiple output vectors to generate an output matrix for determining the language of the text to be identified.

[0073] Optionally, in the model training module 607 of the device, the value inside the masking matrix is ​​0, and the value outside the masking matrix is ​​negative infinity.

[0074] The feature fusion apparatus of this embodiment is used to implement the corresponding feature fusion methods in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the feature fusion apparatus of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0075] In a third aspect of the embodiments of this application, an electronic device is provided, such as Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0076] like Figure 7 As shown, the electronic device may include: a processor 702, a communications interface 704, a memory 706, and a communications bus 708.

[0077] in:

[0078] The processor 702, communication interface 704, and memory 706 communicate with each other via communication bus 708.

[0079] Communication interface 704 is used to communicate with other electronic devices or servers.

[0080] The processor 702 is used to execute program 710, specifically to perform the relevant steps in the above-described feature fusion method embodiment.

[0081] Specifically, program 710 may include program code that includes computer operation instructions.

[0082] The processor 702 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0083] Memory 706 is used to store program 710. Memory 706 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0084] Specifically, program 710 can be used to cause processor 702 to perform the following operations:

[0085] Obtain the text to be recognized, and determine the character order and text length of the characters contained in the text to be recognized;

[0086] The character temporal features, coding block features, word features, and character features of the text to be identified are generated respectively. The order of the feature units contained in the character temporal features, coding block features, word features, and character features is the same as the character order. The length of the character temporal features, coding block features, word features, and character features is the same as the length of the text.

[0087] The character temporal features, coded block features, word features, and character features are fused to generate a multi-feature representation vector.

[0088] The specific implementation of each step in program 710 can be found in the corresponding steps and units described in the above-described feature fusion method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the aforementioned method embodiments, and will not be repeated here.

[0089] In a fourth aspect of this application, a computer storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the feature fusion method as described in the first aspect.

[0090] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0091] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored as software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the feature fusion method described herein. Furthermore, when a general-purpose computer accesses code used to implement the feature fusion method shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the feature fusion method shown herein.

[0092] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0093] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.< / bl> < / bl> < / bl> < / ls>

Claims

1. A feature fusion method for language recognition, comprising: Obtain the text to be recognized, and determine the character order and text length of the characters contained in the text to be recognized; The character temporal features, coding block features, word features, and character features of the text to be identified are generated respectively. The order of the feature units contained in the character temporal features, coding block features, word features, and character features is the same as the character order. The length of the character temporal features, coding block features, word features, and character features is the same as the length of the text. The character temporal features are used to characterize the character order of the characters contained in the text to be identified. The character temporal features, coded block features, word features and character features are fused to generate a multi-feature representation vector. The original information matrix and multiple mask matrices corresponding to the multi-feature representation vector are used for model training. Different mask matrices contain matrix window masks of different sizes.

2. The method of claim 1, wherein, The multi-dimensional vector is generated by fusing the character temporal features, coded block features, word features, and character features, including: The character temporal features, coded block features, word features, and character features are summed to generate a multi-feature representation vector; or, The character temporal features, coded block features, word features, and character features are concatenated to generate a multi-feature representation vector.

3. The method of claim 2, wherein, The multi-feature representation vector is generated by fusing the character temporal features, coded block features, word features, and character features, including: Perform a linear or nonlinear transformation on the generated multi-feature representation vector to obtain the transformed multi-feature representation vector.

4. The method of claim 1, wherein, The word features of the text to be identified are generated in the following manner; Determine the sub-words contained in the words of the text to be identified and their order, wherein the sub-words include prefixes, suffixes or roots contained in the words; The word is segmented according to its subwords, and word features of the text to be identified are generated according to the order of the subwords.

5. The method of claim 1, wherein, The method further includes: Determine the transpose vector of the multi-feature representation vector, and multiply the transpose vector of the multi-feature representation vector with the multi-feature representation vector to generate the original information matrix; Determine multiple masking matrices of the same size as the original information matrix, wherein the multiple masking matrices contain matrix window masks of different sizes; By adding the original information matrix and the multiple masking matrices respectively, multiple feature matrices are obtained; The multiple feature matrices are normalized to obtain multiple normalized feature matrices, and the model is trained based on the multiple normalized feature matrices.

6. The method of claim 5, wherein, The plurality of masking matrices contain matrix window masks of different sizes, including: The size of the matrix window mask in the N masking matrices increases sequentially from 1*1 to N*N, where N is a positive integer less than the length of the text; The size of the matrix window mask in the remaining masking matrices is equal to the size of the original information matrix.

7. The method of claim 5, wherein, The plurality of masking matrices contain matrix window masks of different sizes, including; Determine the word length of the words contained in the text to be identified; Determine the matrix window mask that matches the word length in the masking matrix.

8. The method of claim 5, wherein model training is performed based on the plurality of normalized feature matrices, comprising: The multiple normalized feature matrices are multiplied by the transpose of the multiple feature representation vectors to generate multiple corresponding output vectors. The multiple output vectors are merged to generate an output matrix, which is used to determine the language of the text to be identified.

9. The method of claim 5, wherein, In the masking matrix, the value inside the matrix window mask is 0, and the value outside the matrix window mask is negative infinity.

10. A feature fusion device for language recognition, comprising: The module determines the text to be identified, and determines the character order and text length of the characters contained in the text to be identified. The feature generation module generates character temporal features, encoding block features, word features, and character features of the text to be identified, respectively. The order of the feature units contained in the character temporal features, encoding block features, word features, and character features is the same as the character order. The length of the character temporal features, encoding block features, word features, and character features is the same as the length of the text. The character temporal features are used to characterize the character order of the characters contained in the text to be identified. The fusion module fuses the character temporal features, coded block features, word features, and character features to generate a multi-feature representation vector. The original information matrix and multiple mask matrices corresponding to the multi-feature representation vector are used for model training. Different mask matrices contain matrix window masks of different sizes.

11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the feature fusion method as described in any one of claims 1-9.

Images