An information fusion method and apparatus

Abstract

The application discloses an information fusion method and device, and relates to the technical field of computers. A specific implementation of the method comprises: performing word embedding on words of a text to be processed, performing position embedding on position information of the words in the text to be processed, obtaining corresponding word vectors and position vectors; performing feature fusion on the word vectors and the position vectors, obtaining original fusion features, and extracting a convolution kernel corresponding to the position information according to the position vectors; performing convolution on the original fusion features using the convolution kernel, obtaining local fusion features, and obtaining new word vectors fused with the position information based on the local fusion features. The implementation extracts local information, realizes local perception and convolution kernel sharing, and then obtains new word vectors fused with the position information by using the local information, so that the position information and the word information can be efficiently and sufficiently fused.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to an information fusion method and device. BACKGROUND

[0002] Transformer is a neural network based on self-attention proposed by Google team, which is a core module for solving natural language processing (NLP) problems. The current mature pre-training language model, such as BERT and Roberta, is designed based on Transformer.

[0003] When modeling text sequence based on Transformer, the position information of words in the text will participate in self-attention calculation, but Transformer does not have the modeling ability of position information, which leads to the fact that position information and word information cannot be fully fused in self-attention calculation. In order to solve this problem, the existing technology mainly uses three ways of absolute position encoding, relative position encoding and separate position encoding to model the text sequence.

[0004] In the process of implementing the present application, there are at least the following problems in the prior art:

[0005] In the self-attention calculation, the above three ways of fusing position information and word information are not sufficient and have low fusion efficiency. SUMMARY

[0006] Therefore, the embodiments of the present application provide an information fusion method and device, which first extracts a position-related convolution kernel using a position vector corresponding to a to-be-processed text, and then convolves the original fusion feature using the convolution kernel to extract local information, realize local perception and convolution kernel sharing, and further obtain a new word vector fused with position information using the local information, so that the position information and the word information can be efficiently and sufficiently fused.

[0007] To achieve the above object, according to an aspect of the embodiments of the present application, an information fusion method is provided.

[0008] The information fusion method of the embodiments of the present application comprises: performing word embedding on words of a to-be-processed text, and performing position embedding on position information of the words in the to-be-processed text to obtain corresponding word vectors and position vectors; performing feature fusion on the word vectors and the position vectors to obtain original fusion features, and extracting a convolution kernel corresponding to the position information according to the position vectors; convolving the original fusion features using the convolution kernel to obtain local fusion features, and obtaining a new word vector fused with the position information based on the local fusion features.

[0009] Optionally, the extracting the convolution kernel corresponding to the position information according to the position vector comprises: performing linear transformation on the position vector to obtain a first transformation result; performing nonlinear transformation on the first transformation result by using a first activation function to obtain a second transformation result; and mapping the second transformation result to a set interval by using a second activation function to obtain the convolution kernel corresponding to the position information of the position vector.

[0010] Optionally, the performing convolution on the original fusion feature by using the convolution kernel to obtain a local fusion feature comprises: performing convolution on the original fusion feature corresponding to one or more input channels respectively by using a depth separable convolution to obtain the local fusion feature corresponding to the input channel; wherein the multiple input channels share the convolution kernel.

[0011] Optionally, the obtaining a new word vector fused with the position information based on the local fusion feature comprises: performing linear transformation on a local fusion feature set corresponding to a word of the text to be processed to obtain the new word vector fused with the position information; wherein the local fusion feature set comprises the local fusion feature corresponding to the one or more input channels.

[0012] Optionally, the performing feature fusion on the word vector and the position vector to obtain an original fusion feature comprises: splicing the word vector and the position vector to obtain a spliced vector; and performing shallow feature fusion on the spliced vector by using a Pointwise convolution to obtain the original fusion feature.

[0013] Optionally, the first activation function is an unsaturated nonlinear activation function, and the second activation function is a softmax function.

[0014] Optionally, the method further comprises: inputting the new word vector into a pre-trained language model to obtain a semantic representation of a word in the text to be processed by a Transformer structure in the pre-trained language model.

[0015] To achieve the above object, according to another aspect of the embodiment of the present application, an information fusion device is provided.

[0016] The information fusion device of the embodiment of the application comprises: an embedding module configured to perform word embedding on words of a to-be-processed text, and position embedding on position information of the words in the to-be-processed text, to obtain corresponding word vectors and position vectors; an extraction module configured to perform feature fusion on the word vectors and the position vectors to obtain original fusion features, and extract a convolution kernel corresponding to the position information according to the position vectors; and a fusion module configured to perform convolution on the original fusion features using the convolution kernel to obtain local fusion features, and obtain new word vectors fused with the position information based on the local fusion features.

[0017] Optionally, the extraction module is further configured to perform linear transformation on the position vectors to obtain first transformation results, perform non-linear transformation on the first transformation results using a first activation function to obtain second transformation results, and map the second transformation results to a set interval using a second activation function to obtain the convolution kernel corresponding to the position information of the position vectors.

[0018] Optionally, the fusion module is further configured to perform convolution on original fusion features corresponding to one or more input channels respectively using a depth separable convolution to obtain local fusion features corresponding to the input channels, wherein the input channels share the convolution kernel.

[0019] Optionally, the fusion module is further configured to perform linear transformation on a set of local fusion features corresponding to the words of the to-be-processed text to obtain new word vectors fused with the position information, wherein the set of local fusion features comprises the local fusion features corresponding to the one or more input channels.

[0020] Optionally, the extraction module is further configured to concatenate the word vectors and the position vectors to obtain a concatenated vector, and perform shallow feature fusion on the concatenated vector using a Pointwise convolution to obtain the original fusion features.

[0021] Optionally, the first activation function is an unsaturated non-linear activation function, and the second activation function is a softmax function.

[0022] Optionally, the device further comprises an input module configured to input the new word vectors into a pre-trained language model, and obtain semantic representations of the words in the to-be-processed text through a Transformer structure in the pre-trained language model.

[0023] To achieve the above object, according to another aspect of the embodiment of the application, an electronic device is provided.

[0024] An electronic device includes one or more processors; and a storage storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement an information fusion method.

[0025] To achieve the above object, according to another aspect of the embodiments of the present application, a computer readable medium is provided.

[0026] A computer readable medium according to an embodiment of the present application has a computer program stored thereon, the program being executed by a processor to implement an information fusion method according to an embodiment of the present application.

[0027] An embodiment of the above application has the following advantages or beneficial effects: first, a position-related convolution kernel is extracted using a position vector corresponding to a to-be-processed text, then the original fusion feature is convolved using the convolution kernel to extract local information, realize local perception and convolution kernel sharing, and further obtain a new word vector with position information using the local information, which can efficiently and sufficiently fuse position information and word information.

[0028] For each position vector, a nonlinear transformation is first performed, and then an interval mapping is performed on the transformed result, so that the position-related convolution kernel can be obtained using the position vector, which facilitates subsequent extraction of local information and reduces the amount of calculation. The deep separable convolution is used to convolve each input channel to ensure that the word vector and the position vector can be sufficiently fused. By performing linear transformation on the local fusion feature set, it is further ensured that a new word vector with position information can be obtained.

[0029] By using Pointwise convolution, shallow feature fusion is performed on the spliced vector of the word vector and the position vector, which further reduces the amount of calculation while ensuring the fusion effect. The input data is mapped to the output end by using an unsaturated nonlinear activation function and a softmax function, which facilitates subsequent processing. The obtained new word vector is input into a pre-trained language model for internal self-attention calculation, which realizes text sequence modeling combined with position information.

[0030] The further effects of the above-mentioned non-conventional optional mode will be described in conjunction with the specific embodiments below. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings serve to better understand the present application and do not constitute an improper limitation thereof. Among them:

[0032] Figure 1 is a schematic diagram of the main steps of the information fusion method according to an embodiment of the present application;

[0033] Figure 2is a schematic diagram of the implementation principle of an information fusion method according to an embodiment of the present application;

[0034] Figure 3 is a schematic diagram of the main flow of an information fusion method according to an embodiment of the present application;

[0035] Figure 4 is a schematic diagram of the main modules of an information fusion device according to an embodiment of the present application;

[0036] Figure 5 is an exemplary system architecture diagram to which embodiments of the present application can be applied;

[0037] Figure 6 is a structural schematic diagram of a computer device suitable for an electronic device to implement embodiments of the present application. DETAILED DESCRIPTION

[0038] Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding, which should be considered in their context only. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0039] In conjunction with the background, the core structure of the Transformer is self-attention, and its calculation process involves three elements of Query (Q), Key (K) and Value (V). The input representation vector will first be transformed by a matrix to obtain the corresponding Q, K and V. Currently, the language model based on the Transformer mainly realizes the modeling of the text sequence through the absolute position encoding, relative position encoding or separate position encoding.

[0040] However, in the absolute position encoding mode, the position vector and the word vector share the calculation parameters, which causes the attention score obtained by the self-attention calculation to be unable to efficiently fuse the position information and the word information. In the relative position encoding mode, only the relative position information is simply used to enhance the representation of the key value in the self-attention calculation, and it is still difficult to efficiently obtain the deep fusion information between the position vector and the word vector. In the separate position encoding mode, although the calculation parameters are not shared, the calculation parameters are completely randomized and irrelevant to the input information, which is not conducive to modeling the association relationship between the position information and the word information.

[0041] In order to solve the defects of insufficient fusion and low fusion efficiency of the position information and the word information in the above three modes, the embodiments of the present application provide a new information fusion method, which can efficiently and sufficiently fuse the position information and the word information, and the specific implementation process is as follows.

[0042] Figure 1 is a schematic diagram of main steps of the information fusion method according to an embodiment of the present application.

[0043] As shown in Figure 1 , the information fusion method according to an embodiment of the present application mainly comprises the following steps:

[0044] Step S101: performing word embedding on words of a to-be-processed text, and performing position embedding on position information of the words in the to-be-processed text to obtain corresponding word vectors and position vectors. Word embedding is mapping words to high-dimensional vectors, which can be implemented in a manner such as Word2Vec (a tool for converting words into vector form), FastText (a tool for word vector calculation and text classification), etc. Position embedding is mapping position information to high-dimensional vectors.

[0045] In an embodiment, the to-be-processed text is first subjected to word segmentation processing to obtain a word sequence, then each word of the word sequence is subjected to word embedding to obtain word vectors corresponding to the words, and position information of each word in the to-be-processed text is subjected to position embedding to obtain position vectors corresponding to the position information.

[0046] Step S102: performing feature fusion on the word vectors and the position vectors to obtain original fusion features, and extracting a convolution kernel corresponding to the position information according to the position vectors. In this step, the word vectors and the position vectors are subjected to feature fusion in a convolution manner to obtain the original fusion features. And for each position vector, a position-related convolution kernel is obtained in a manner of nonlinear transformation and interval mapping.

[0047] Step S103: performing convolution on the original fusion features using the convolution kernel to obtain local fusion features, and obtaining new word vectors with the position information fused based on the local fusion features. The convolution kernel obtained in step S102 is used to perform convolution on at least one input channel to obtain local fusion features with local word information and position information fused; subsequent linear transformation on the local fusion features can obtain new word vectors with the position information.

[0048] This embodiment realizes local perception and convolution kernel sharing, and can efficiently and sufficiently fuse position information and word information.

[0049] Figure 2 is a schematic diagram of an implementation principle of the information fusion method according to an embodiment of the present application. As Figure 2As shown, the text to be processed in this embodiment contains n words. First, word embedding is performed on each of the n words to obtain the corresponding word vectors. Then, position embedding is performed on each of the n words at their respective positions in the text to be processed to obtain the corresponding position vectors. After that, position-related convolution kernels are extracted based on the position vectors. Finally, using the convolution kernels, depthwise separable convolution is performed on the original fused features corresponding to the word vectors and position vectors to obtain new word vectors with position information for the n words.

[0050] For ease of explanation, the n words can be represented as X = {x1, x2, ..., xn}. n The word vectors corresponding to n words can be represented as W = {w1, w2, ..., w...} n The position vectors corresponding to n words can be represented as P = {p1, p2, ..., p...} n The new word vectors corresponding to n words can be represented as W'={w'1,w'2,……,w' n ].

[0051] Figure 3 This is a schematic diagram of the main flow of the information fusion method according to an embodiment of the present invention. Figure 3 As shown, the information fusion method of this invention mainly includes the following steps:

[0052] Step S301: Perform word embedding on the words in the text to be processed to obtain the word vectors corresponding to each word. The text to be processed is segmented to obtain a word sequence, and then word embedding is performed on each word in the word sequence to obtain the word vectors corresponding to each word. Assuming the text to be processed is: "How much does this phone cost?", the word sequence obtained from word segmentation could be: "this", "each", "phone", "how much", "money".

[0053] Suppose the text to be processed contains n words, represented as X = [x1, x2, ..., xn]. n}, where X∈R n X is an n-dimensional vector, where n is an integer and each dimension of the vector has a real number value. Word embedding is performed on each of the n words to obtain the corresponding word vector W = {w1, w2, ..., wn}. n}, where w i ∈R d , 1≤i≤n, d is the embedding dimension.

[0054] Step S302: Embed the position information of words in the text to be processed to obtain the position vector corresponding to each position. Encode the position information of n words appearing at n positions in the text to be processed to obtain the corresponding position vector P = {p1, p2, ..., p...} n}, where p i ∈Rd d is the embedding dimension, and its value is the same as the embedding dimension of word embedding in step S301.

[0055] Step S303: Fuse the word vectors and position vectors of corresponding words to obtain the original fused features. Concatenate the word vectors and position vectors of corresponding words to obtain the concatenated vectors for each word. Then, use pointwise convolution to perform shallow feature fusion on the concatenated vectors to obtain the original fused features for each word. The shallow features are the embedding layers of the deep learning network, or the output vectors of the first and second layers of the network.

[0056] Pointwise (PW) convolution has a kernel size of 1×1×M, where M is the number of channels in the previous layer. This convolution operation weights and combines the feature maps from the previous step along the depth direction to generate new feature maps. The number of output feature maps corresponds to the number of convolution kernels. This convolution has a learnable parameter W. bottle .

[0057] The expression for the original fused feature corresponding to one of the n words is as follows:

[0058]

[0059] In the formula, h i ∈R 2d , represents the original fusion feature of the i-th word in the text to be processed; 2d represents the number of input channels, and c represents the current input channel; The learnable parameters for pointwise convolution; g[w i ;p i ] is the concatenation vector corresponding to the i-th word of the text to be processed.

[0060] Based on Formula 1, the original fusion features corresponding to n words can be represented as H = {h1, h2, ..., h...} n}∈R n×2d .

[0061] Step S304: Extract the convolution kernel corresponding to each position information based on its position vector. In this step, to extract position-specific convolution kernels, the position vectors of each position information are first linearly transformed to obtain a first transformation result; then, a first activation function is used to perform a nonlinear transformation on the first transformation result to obtain a second transformation result; finally, a second activation function is used to map the second transformation result to a set interval to obtain the convolution kernel corresponding to each position information.

[0062] In the embodiment, the first activation function is an unsaturated nonlinear activation function, such as a Relu function, a GELU function; and the second activation function is a softmax function. When mapping the second transformation result to a set interval, the softmax operation is performed on the second transformation result along the embedding dimension d, so that the second transformation result can be mapped to the interval (0, 1). In this way, the convolution kernel obtained at each position is different and closely related to the position information.

[0063] wherein the expression of the second transformation result is as follows:

[0064]

[0065] wherein, is the second transformation result corresponding to the i th word of the text to be processed; W p1 , b p1 , W p2 and b p2 are learnable parameters, which are used for projection transformation of the feature representation result in a high-dimensional space and the like. Among them, b p2 ∈ R k ; d1 = γ × d; γ ∈ R, γ is greater than 1, and is preferably 4.

[0066] The expression of the convolution kernel is as follows:

[0067]

[0068] wherein, is the convolution kernel corresponding to the i th word of the text to be processed, and the size of the convolution kernel is k.

[0069] Step S305: using the convolution kernel corresponding to the same word to convolve the original fusion feature to obtain a set of local fusion features corresponding to each word. Based on the convolution kernel obtained above, a depth separable convolution is used to convolve the original fusion feature corresponding to one or more input channels along the dimension n of the word sequence to obtain a local fusion feature corresponding to the input channel. The plurality of input channels share the convolution kernel.

[0070] The depth separable convolution (Depthwise Separable Convolution) is combined by the Depthwise (DW) convolution and the Pointwise (PW) convolution, which can be used to extract features. Compared with the conventional convolution operation, the depth separable convolution has a lower number of parameters and a lower operation cost.

[0071] In the embodiment, the convolution kernel shared by the input channels 1-2d is Finally, for each input channel, a local fusion feature O i,cAmong them, O i,c It can be expressed by the following formula:

[0072]

[0073] In the formula, O i,c represents the local fusion feature of the i-th word and c-th input channel of the text to be processed; k is the convolution kernel size; The convolution kernel at position j of the i-th word in the text to be processed (i.e., a convolution kernel of length k). The value at the j-th position); Let be the original fusion features of the i-th word and c-th input channel of the text to be processed; 1≤c≤2d.

[0074] Formula 4 indicates that for the i-th word in the text to be processed, the 2d feature dimensions share the same convolutional kernel. When calculating the output corresponding to the i-th word, a window of size k needs to be selected around i, i.e., [i-(k+1) / 2, i+(k+1) / 2], and a convolution kernel is used. Calculate the output of the local receptive field.

[0075] Based on the above formula, the local fusion features corresponding to all input channels of the i-th word in the text to be processed constitute the local fusion feature set. Therefore, the local fusion feature set corresponding to the i-th word in the text to be processed can be O(n). i This indicates that its value is 0. i ={O i,1 O i,2 ,……,O i,2d}∈R 2d The local fusion feature set corresponding to n words in the text to be processed can be represented by O, and its value is O = {O1, O2, ..., O2}. n}∈R n×2d .

[0076] Step S306: Perform a linear transformation on the local fusion feature set corresponding to each word to obtain a new word vector with fused positional information. The local fusion features corresponding to all input channels of the n words in the text to be processed constitute the local fusion feature set. Performing a linear transformation on this local fusion feature set yields a new word vector with positional information.

[0077] Wherein, the new word vector w' corresponds to the i-th word in the text to be processed. i The expression is as follows:

[0078] w' i =W O O i +b O Formula 5

[0079] In the formula, w' i ∈R d , 1≤i≤n; W O and b O W is a learnable parameter of the linear transformation. O ∈R d×2d b O This is the offset.

[0080] It should be noted that this embodiment does not limit the execution order of the above steps. Among these steps, steps S301 and S302 can be executed sequentially or in parallel; steps S303 and S304 can also be executed sequentially or in parallel; and step S303 must be executed after steps S301 and S302 have been executed, and step S304 must be executed after step S302 has been executed.

[0081] In a preferred embodiment, after obtaining the new word vectors, the new word vectors can also be input into a pre-trained language model, and the semantic representation of the words in the text to be processed can be obtained through the Transformer structure in the pre-trained language model.

[0082] As can be seen from the information fusion method of this embodiment, this embodiment first extracts position-related convolution kernels using the position vectors corresponding to the text to be processed, and then uses depthwise separable convolution to fuse and encode word information and position information, extracting local information, realizing local perception and convolution kernel sharing of input channels, and then using local information to obtain new word vectors fused with position information. Under the premise of ensuring fusion effect, the amount of computation is reduced, and position information and word information can be fused efficiently and fully.

[0083] Figure 4 This is a schematic diagram of the main modules of the information fusion device according to an embodiment of the present invention.

[0084] like Figure 4 As shown, the information fusion device 400 of this embodiment mainly includes:

[0085] The embedding module 401 is used to embed words in the text to be processed, and to embed the position information of the words in the text to be processed to obtain corresponding word vectors and position vectors. The text to be processed is segmented to obtain a word sequence, and then word embedding is performed on each word in the word sequence to obtain the corresponding word vector, and the position information of each word in the text to be processed is embedded to obtain the corresponding position vector.

[0086] The extraction module 402 is used to perform feature fusion on the word vectors and the position vectors to obtain the original fused features, and to extract the convolutional kernels corresponding to the position information based on the position vectors. Convolution is used to fuse the word vectors and position vectors to obtain the original fused features. Furthermore, for each position vector, a nonlinear transformation and interval mapping are used to obtain position-related convolutional kernels.

[0087] The fusion module 403 is used to convolve the original fusion features using the convolution kernel to obtain local fusion features, and to obtain a new word vector with the fused positional information based on the local fusion features. The convolution kernel obtained by the extraction module 402 is used to convolve at least one input channel to obtain local fusion features with local word information and positional information; subsequently, a linear transformation is performed on the local fusion features to obtain a new word vector with positional information.

[0088] In addition, the information fusion device 400 of this embodiment may further include: an input module ( Figure 4 (Not shown in the text) This module is used to input the new word vectors into a pre-trained language model, and obtain the semantic representation of the words in the text to be processed through the Transformer structure in the pre-trained language model.

[0089] As can be seen from the above description, the location-related convolution kernels are first extracted using the location vectors corresponding to the text to be processed. Then, the convolution kernels are used to convolve the original fused features to extract local information, realize local perception and convolution kernel sharing, and then use the local information to obtain new word vectors that are fused with location information. This can efficiently and fully fuse location information and word information.

[0090] Figure 5 An exemplary system architecture 500 is shown that can be applied to the information fusion method or information fusion apparatus of the present invention.

[0091] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0092] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Terminal devices 501, 502, and 503 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0093] The server 505 can be a server providing various services, for example, a background management server processing the to-be-processed text sent by the administrator using the terminal device 501, 502, 503. The background management server can perform word embedding, position embedding, feature fusion, extraction of convolution kernel, convolution operation and the like on the to-be-processed text, and feed back the processing result (for example, the obtained new word vector) to the terminal device.

[0094] It should be noted that the information fusion method provided by the embodiment of the present application is generally executed by the server 505, and correspondingly, the information fusion device is generally arranged in the server 505.

[0095] It should be understood that, Figure 5 The number of terminal devices, networks and servers in the above-mentioned system is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

[0096] According to the embodiment of the present application, the present application also provides an electronic device and a computer readable medium.

[0097] The electronic device of the present application comprises one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement an information fusion method of an embodiment of the present application.

[0098] The computer readable medium of the present application has a computer program stored thereon, which is executed by a processor to implement an information fusion method of an embodiment of the present application.

[0099] The following Figure 6 shows the structure of a computer system 600 suitable for implementing the electronic device of the embodiment of the present application. Figure 6 The electronic device shown is only an example, and should not bring any limitation to the function and use range of the embodiment of the present application.

[0100] As shown in Figure 6 , the computer system 600 comprises a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the programs stored in the read-only memory (ROM) 602 or the programs loaded from the storage part 608 to the random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602 and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0101] The following components are connected to the I / O interface 605: an input part 606 including a keyboard, a mouse, etc.; an output part 607 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 608 including a hard disk, etc.; and a communication part 609 including a network interface card such as a LAN card, a modem, etc. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 610 as necessary, so that a computer program read out therefrom is installed in the storage part 608 as necessary.

[0102] In particular, according to the embodiments of the present disclosure, the process described in the above main flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable medium, the computer program containing program codes for executing the method shown in the main flowchart. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-described functions defined in the system of the present disclosure are executed.

[0103] It should be noted that the computer-readable medium shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present application, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0104] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0105] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The described modules can also be arranged in a processor, for example, a processor can be described as including an embedding module, an extracting module and a fusion module. In some cases, the names of these modules do not constitute a limitation on the modules themselves, for example, the embedding module can also be described as "a module that performs word embedding on a word of a to-be-processed text, performs position embedding on position information of the word in the to-be-processed text, and obtains a corresponding word vector and a position vector".

[0106] As another aspect, the present application also provides a computer readable medium, which can be included in the device described in the above embodiments, or can exist independently without being assembled into the device. The computer readable medium carries one or more programs, which, when executed by the device, cause the device to include: performing word embedding on a word of a to-be-processed text, performing position embedding on position information of the word in the to-be-processed text, and obtaining a corresponding word vector and a position vector; performing feature fusion on the word vector and the position vector to obtain original fusion features, and extracting a convolution kernel corresponding to the position information according to the position vector; performing convolution on the original fusion features using the convolution kernel to obtain local fusion features, and obtaining a new word vector fused with the position information based on the local fusion features.

[0107] According to the technical solution of the embodiments of the present application, the position-related convolution kernel is first extracted using the position vector corresponding to the to-be-processed text, and then the original fusion features are convolved using the convolution kernel to extract local information, realize local perception and convolution kernel sharing, and further obtain a new word vector fused with the position information using the local information, which can efficiently and sufficiently fuse the position information and the word information.

[0108] The above product can execute the method provided by the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the embodiments can be referred to the method provided by the embodiments of the present application.

[0109] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made depending on design requirements and other factors. Any modification, equivalent replacement and improvement made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An information fusion method, characterized in that, include: Word embedding is performed on the words in the text to be processed, and position embedding is performed on the position information of the words in the text to be processed to obtain the corresponding word vectors and position vectors; The word vector and the position vector are fused to obtain the original fused features, and the convolution kernel corresponding to the position information is extracted based on the position vector. The original fused features are convolved using the convolution kernel to obtain local fused features, and a new word vector incorporating the location information is obtained based on the local fused features.

2. The method according to claim 1, characterized in that, The step of extracting the convolution kernel corresponding to the position information based on the position vector includes: Perform a linear transformation on the position vector to obtain the first transformation result; The first transformation result is subjected to a nonlinear transformation using a first activation function to obtain a second transformation result. A second activation function is used to map the second transformation result to a set interval to obtain the convolution kernel corresponding to the position information of the position vector.

3. The method according to claim 1, characterized in that, The step of convolving the original fused features with the convolution kernel to obtain local fused features includes: Depthwise separable convolution is used to convolve the original fusion features corresponding to one or more input channels respectively to obtain the local fusion features corresponding to the input channels; wherein the multiple input channels share the convolution kernel.

4. The method according to claim 3, characterized in that, The process of obtaining a new word vector incorporating the location information based on the local fusion features includes: A linear transformation is performed on the local fusion feature set corresponding to the words in the text to be processed to obtain a new word vector fused with the positional information; wherein, the local fusion feature set includes the local fusion features corresponding to one or more input channels.

5. The method according to claim 1, characterized in that, The feature fusion of the word vector and the position vector to obtain the original fused features includes: The word vector and the position vector are concatenated to obtain the concatenated vector; Pointwise convolution is used to perform shallow feature fusion on the concatenated vectors to obtain the original fused features.

6. The method according to claim 2, characterized in that, The first activation function is an unsaturated nonlinear activation function, and the second activation function is a softmax function.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The new word vectors are input into a pre-trained language model, and the semantic representations of the words in the text to be processed are obtained through the Transformer structure in the pre-trained language model.

8. An information fusion device, characterized in that, include: The embedding module is used to embed words in the text to be processed, and to embed the position information of the words in the text to be processed to obtain the corresponding word vectors and position vectors. An extraction module is used to perform feature fusion on the word vector and the position vector to obtain the original fused features, and to extract the convolution kernel corresponding to the position information based on the position vector; The fusion module is used to convolve the original fusion features using the convolution kernel to obtain local fusion features, and to obtain a new word vector that incorporates the location information based on the local fusion features.

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images