Feature fusion method and device, computer equipment and storage medium
The feature fusion method addresses the limitations of existing methods by incorporating importance and correlation in feature representations, resulting in improved feature interaction and representation ability.
Patent Information
- Application Number
- CN202410711537.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-03
- Publication Date
- 2024-12-17
AI Technical Summary
Existing multimodal feature fusion methods, such as bilinear pooling, fail to accurately capture the interactivity and dependency between different data features, leading to poor feature representation ability and expression in fused feature representations.
A feature fusion method that utilizes a superposition of first and second feature representations, where the first representation captures the importance of data features during interaction and the second representation captures their correlation, enabling a more comprehensive fusion that considers both importance and correlation.
The method improves the richness and expressiveness of the final fused feature representation by accurately describing the interactivity between data features, enhancing the feature representation ability.
Smart Images
Figure CN119150215A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a feature fusion method, device, computer equipment and storage medium (computer-readable storage medium). Background Art
[0002] Multimodal feature fusion refers to the integration of features from different modalities, such as images, audio, and text, to extract more comprehensive and accurate information. In multimodal tasks, deep learning models are usually used to extract feature representations of different modalities. In multimodal feature fusion, a common method is to use a bilinear pooling operation to multiply two feature matrices and obtain attention weights through a fully connected layer. The attention weights are applied to the feature representation of each modality to weightedly fuse information from multiple modalities. Summary of the invention
[0003] Embodiments of the present application provide a feature fusion method, apparatus, computer device, and storage medium.
[0004] In a first aspect, an embodiment of the present application provides a feature fusion method, including:
[0005] Acquire a feature set; the feature set includes at least two types of data features;
[0006] Based on the feature information of the data features included in the feature set, a first feature representation and a second feature representation are obtained; the first feature representation represents the importance of the data features included in the feature set during feature interaction; the second feature representation represents the association relationship between the data features included in the feature set;
[0007] The first feature representation and the second feature representation are fused to obtain a fused feature representation.
[0008] In a second aspect, an embodiment of the present application provides a feature fusion device, including:
[0009] A feature acquisition module, used to acquire a feature set; the feature set includes at least two types of data features;
[0010] A first feature fusion module, configured to fuse feature information of the data features included in the feature set to obtain a first feature representation and a second feature representation; the first feature representation represents the importance of the data features included in the feature set during feature interaction; the second feature representation represents the association relationship between the data features included in the feature set;
[0011] The second feature fusion module is used to fuse the first feature representation and the second feature representation to obtain a fused feature representation.
[0012] In a third aspect, an embodiment of the present application further provides a computer device, comprising a memory storing a plurality of computer programs; a processor loads the computer program from the memory to execute any one of the feature fusion methods provided in the embodiment of the present application.
[0013] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a plurality of computer programs, and the computer programs are suitable for loading by a processor to execute any feature fusion method provided in the embodiment of the present application.
[0014] In a fifth aspect, an embodiment of the present application also provides a computer program product, including a computer program or a computer program, which implements any feature fusion method provided in the embodiment of the present application when the computer program or the computer program is executed by a processor.
[0015] By adopting the scheme of the application embodiment, the importance of each data feature during feature interaction is captured through the first feature representation, and the correlation between each data feature is captured through the second feature representation, so that the correlation, mutual influence and dependency between each data feature can be determined. Therefore, after the first feature representation is fused and added to the second feature representation, the final fused feature representation obtained not only considers the correlation between the data features, but also considers the importance of the data features during feature interaction, so that the final fused feature representation more accurately and comprehensively describes the relationship between each data feature, and improves the richness, expression ability and expressiveness of the final fused feature representation. Therefore, it can more accurately describe the interactivity between each data feature, thereby improving the feature representation ability of the final fused feature representation. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0017] Figure 1 It is a schematic diagram of the application environment of the feature fusion method provided in the embodiment of the present application;
[0018] Figure 2 It is a flowchart of the feature fusion method provided in the embodiment of the present application;
[0019] Figure 3 It is a flowchart of the data feature extraction process provided in the embodiment of the present application;
[0020] Figure 4It is a flowchart of the feature information extraction process provided in the embodiment of the present application;
[0021] Figure 5 It is a flowchart of the feature information fusion process provided in the embodiment of the present application;
[0022] Figure 6 It is a schematic diagram of a first characteristic matrix provided in an embodiment of the present application;
[0023] Figure 7 is a schematic diagram of a second characteristic matrix provided in an embodiment of the present application;
[0024] Figure 8 It is a schematic diagram of a third characteristic matrix provided in an embodiment of the present application;
[0025] Fig. 9 is a schematic diagram of a fourth characteristic matrix provided in an embodiment of the present application;
[0026] Fig.10 is a schematic diagram of the fusion feature representation provided in the embodiment of the present application;
[0027] Fig.11 is a schematic diagram of the structure of a feature fusion device provided in an embodiment of the present application;
[0028] Fig.12 It is a schematic diagram of the internal structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application. At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0030] In one embodiment of the present application, the feature fusion method can be run on a local terminal device or a server. When the feature fusion method is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.
[0031] In order to better understand the feature fusion method, device, computer equipment and storage medium provided in the embodiments of the present application, the application environment applicable to the embodiments of the present application is described below.
[0032] See also Figure 1 , Figure 1 A schematic diagram of an application environment of the feature fusion method provided in an embodiment of the present application is shown. As an implementation mode, the feature fusion method provided in the embodiment of the present application can be applied to an electronic device, wherein the electronic device can be such as Figure 1 The server 110 shown in the figure can be connected to the terminal device 120 through a network. The network is used to provide a medium for a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as a wired communication link, a wireless communication link, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the electronic device can also be a smart phone, a laptop computer, etc.
[0033] It should be understood that Figure 1 The server 110, network and terminal device 120 are merely illustrative. Depending on the implementation requirements, there may be any number of servers, networks and terminal devices. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the terminal device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 at the same time.
[0034] In some embodiments, the terminal device 120 can record the user's voice to obtain the user's audio data. Further, the terminal device 120 sends the user's audio data to the server 110 through the network. After the server 110 receives the user's audio data, the audio data can be processed by the feature fusion method described in the embodiment of the present application.
[0035] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is a terminal device as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.
[0036] The feature fusion method of this embodiment can be applied to multimodal recognition task scenarios, such as sentiment analysis task scenarios, intent analysis task scenarios, etc. In multimodal recognition task scenarios, it is often required to fuse features of at least two types of data (such as text, voice, image, video, etc.) to obtain fused feature representations, and then input the fused feature representations into the recognition task system (such as sentiment analysis system, intent analysis system, etc.) for recognition. In multimodal recognition tasks, deep learning models are usually used to extract feature representations of different modalities. In feature fusion, the existing common method is to use a bilinear aggregation method, that is, to multiply two feature matrices and obtain attention weights through a fully connected layer, so that attention weights can be applied to each data feature representation to weightedly fuse multiple data feature information. Therefore, in the fusion process, the bilinear aggregation method directly weights multiple data feature information, resulting in low interactivity of multiple data feature information, which leads to poor feature interaction expression of the fused feature representation, and reduces the feature representation ability of the fused feature representation.
[0037] In order to solve the above-mentioned technical problems, the inventors of the present application proposed a feature fusion method based on the superposition of a first feature representation and a second feature representation, that is, the importance of each data feature during feature interaction is captured by the first feature representation, and the correlation between each data feature is captured by the second feature representation, so that the correlation, mutual influence and dependency between each data feature can be determined. Therefore, after the first feature representation is fused and added to the second feature representation, the final fused feature representation obtained not only considers the correlation between the data features, but also considers the importance of the data features during feature interaction, so that the final fused feature representation more accurately and comprehensively describes the relationship between each data feature, and improves the richness, expression ability and expressiveness of the final fused feature representation. Therefore, it can more accurately describe the interactivity between each data feature, thereby improving the feature representation ability of the final fused feature representation.
[0038] In some embodiments, the terminal device 120 can respond to user operations (such as clicking on a feature fusion entrance to enter a feature fusion operation page, and entering at least two types of data on the operation page), generate a trigger request for feature fusion, and then perform feature fusion on at least two types of data through the feature fusion method of an embodiment of the present application.
[0039] In other embodiments, the terminal device 120 may generate a trigger request for feature fusion in response to a user operation (such as clicking on a feature fusion entry to enter a feature fusion operation page, and inputting at least two types of data on the operation page), and then send the trigger request to the server 110 through the network. When the server 110 receives the trigger request, it may perform feature fusion on at least two types of data using the feature fusion method of the embodiment of the present application.
[0040] refer to Figure 2 , Figure 2 : is a flow chart of the feature fusion method provided in the embodiment of the present application. The specific process of the feature fusion method can be as follows: Step 10 to Step 30, wherein:
[0041] Step 10: Obtain feature set.
[0042] Optionally, the terminal device responds to the trigger request and obtains at least two types of data, where the at least two types of data include but are not limited to text, voice, video, image, etc.
[0043] Furthermore, the terminal device inputs at least two types of data into the modal backbone network, and performs feature extraction on each type of data of the at least two types of data through the modal backbone network (MBN) to obtain data features corresponding to each type of data, thereby obtaining a feature set corresponding to the at least two types of data. Therefore, it can be understood that the feature set includes at least two types of data features.
[0044] Among them, the modal backbone network is a neural network structure used for processing various types of data. The network structure of the modal backbone network includes multiple network branches, each network branch is responsible for processing a type of data, and the number of network branches is set according to the recognition task scenario. For example, for the sentiment analysis task scenario, the modal backbone network can include a text network branch and a voice network branch. For the intent analysis task scenario, the modal backbone network can include a text network branch, a voice network branch and an image network branch.
[0045] In one embodiment, the recognition task scenario is a sentiment analysis task scenario, the modal backbone network may include a text network branch and a voice network branch, and the at least two types of data input by the user are text data and voice data. For the text network branch, the text data features extracted from the text data may be represented as Ftext m ={(m,N),D m}, where N represents the total number of text data, m is the position of the current text in the total number of text, and D m Indicates the feature dimension corresponding to the text data feature of the current text. For the speech network branch, the speech data feature extracted from the speech data can be represented as Fspeech k ={(k, samples), T k ,d k}, where samples represents the total number of sampling points of speech data, k represents the position of the current speech sampling point in the total number of sampling points, and T kRepresents the time dimension of the speech data features of the current speech sampling point, d k Indicates the feature dimension corresponding to the speech data feature of the current speech sampling point.
[0046] Reference Figure 3 , Figure 3 : is a flow chart of the data feature extraction process provided in the embodiment of the present application, the text data includes text 1, text 2 and text 3, the voice data includes voice sampling point 1, voice sampling point 2, voice sampling point 3 and voice sampling point 4, and the specific feature extraction process is: the terminal device inputs the text data and the voice data into the modal backbone network, and the text network branch extracts features from each text in the text data to obtain the text data feature Ftext of text 1 1 , text data feature Ftext of text 2 2 , the text data feature Ftext of text 3 3 The speech network branch extracts features from each speech sampling point in the speech data and obtains the speech data feature Fspeech of speech sampling point 1. 1 , the speech data feature Fspeech of speech sampling point 2 2 , the speech data feature Fspeech of speech sampling point 3 3 And the speech data feature Fspeech of speech sampling point 4 4 , so we can get the feature set as {Ftext 1 , Ftext 2 , Ftext 3}+{Fspeech 1 , Fspeech 2 , Fspeech 3 , Fspeech 4}.
[0047] Step 20, fusing feature information of the data features included in the feature set to obtain a first feature representation and a second feature representation.
[0048] Optionally, the terminal device extracts feature information of the data features included in the feature set to obtain feature information of the data features included in the feature set.
[0049] In the process of feature information extraction, feature information can be extracted through fully connected layer (FClayer), convolutional neural network (CNN), recurrent neural network (RNN), etc. The feature information of data features can include query vector (Query), key vector (Key) and value vector (Value), wherein the Query query vector is used to find the matching Key key vector, the Key key vector is used to compare the Query query vector, and the Value value vector is used to record the information of the input sequence.
[0050] In an optional embodiment of the present application, the feature set may include a first data feature and a second data feature. Therefore, the process of extracting feature information of the data features included in the feature set is:
[0051] Optional, see Figure 4 , Figure 4 : is a flow chart of the feature information extraction process provided in the embodiment of the present application. The terminal device inputs the first data feature and the second data feature into the fully connected layer (FC layer), and the fully connected layer extracts feature information from the first data feature and the second data feature, and outputs the query vector Q of the first data feature. 1i , key vector K 1i Sum value vector V 1i , where i represents the i-th first data feature, and the query vector Q of the output second data feature 2n , key vector K 2n Sum value vector V 2n , where n represents the nth second data feature.
[0052] In one embodiment, the first data feature is {Ftext 1 , Ftext 2 , Ftext 3}, the second data feature is {Fspeech 1 , Fspeech 2 , Fspeech 3 , Fspeech 4}, the first data feature {Ftext 1 , Ftext 2 , Ftext 3} and the second data feature {Fspeech 1 , Fspeech 2 , Fspeech 3 , Fspeech 4} Input to the fully connected layer to obtain the fully connected layer output data feature Ftext 1 The query vector Q 11 , key vector K 11 Sum value vector V 11 , data feature Ftext 2 The query vector Q 12 , key vector K 12 Sum value vector V 12 , data feature Ftext 2 The query vector Q 13 , key vector K 13 Sum value vector V 13 , data features Fspeech 1 The query vector Q 21 , key vector K 21 Sum value vector V 21 , data features Fspeech 2 The query vector Q 22 , key vector K 22 Sum value vector V 22 , data features Fspeech 3 The query vector Q 23 , key vector K 23 Sum value vector V 23 , data features Fspeech 4 The query vector Q 24 , key vector K 24 Sum value vector V 24 .
[0053] Furthermore, the terminal device fuses the feature information of the data features included in the feature set to obtain a first feature representation and a second feature representation. In one embodiment, the first feature representation is obtained by fusion based on an attention mechanism, so the first feature representation represents the importance of the data features included in the feature set during feature interaction. The second feature representation is obtained by fusion through an interaction degree mechanism, so the second feature representation represents the correlation between the data features included in the feature set.
[0054] Among them, the attention mechanism is a model that simulates the human attention mechanism, which is used to strengthen the deep learning model's attention to different parts of the input data. The interactivity mechanism is usually used to process information interaction in sequence data. In sequence data, there may be complex dependencies between different elements. The interactivity mechanism aims to capture these dependencies in order to better model the association between sequence data. Common interactivity mechanisms include recurrent neural networks (RNNs) and long short-term memory networks (LSTMs). It should be noted that the second feature representation obtained by the interactivity mechanism does not have the property of attention.
[0055] In one embodiment of the present application, the feature information of the data feature includes a query vector, a key vector, and a value vector. Therefore, the specific process of fusing the feature information of the data feature included in the feature set to obtain the first feature representation and the second feature representation includes:
[0056] The query vector, the key vector and the value vector of the data features included in the feature set are fused to obtain a fused vector; and the first feature representation and the second feature representation are determined based on the fused vector.
[0057] Optionally, the terminal device fuses the query vector, key vector and value vector of the data features included in the feature set based on the attention mechanism, and fuses the query vector and key vector of the data features included in the feature set based on the interactivity mechanism to obtain a fused vector. Further, the terminal device fuses the fused vector again to obtain the first feature representation and the second feature representation.
[0058] In one embodiment, the feature set includes a first data feature and a plurality of second data features, and therefore, a query vector, a key vector, and a value vector based on the data features included in the feature set are fused to obtain a fused vector; and a process of determining the first feature representation and the second feature representation based on the fused vector includes:
[0059] Fusing the query vector of the first data feature with the key vectors and value vectors of the plurality of second data features to obtain a first fused vector;
[0060] Fusing the key vector and the value vector of the first data feature with the query vectors of the plurality of second data features to obtain a second fused vector;
[0061] Fusing the query vector of the first data feature with the key vectors of the plurality of second data features to obtain a third fused vector;
[0062] Fusing the key vector of the first data feature with the query vectors of the plurality of second data features to obtain a fourth fused vector;
[0063] The first fused vector is fused with the second fused vector to obtain a first feature representation; and the third fused vector is fused with the fourth vector to obtain a second feature representation.
[0064] Optionally, the terminal device fuses the query vector of the first data feature with the key vector and the value vector of each second data feature in multiple second data features through an attention mechanism to obtain a first fused vector. At the same time, the terminal device fuses the key vector and the value vector of the first data feature with the query vector of each second data feature in multiple second data features through an attention mechanism to obtain a second fused vector. Finally, the first fused vector is fuses with the second fused vector to obtain the first feature representation.
[0065] Optionally, in an optional embodiment of the present application, refer to Figure 5 , Figure 5 is a flow chart of the feature information fusion process provided in the embodiment of the present application, wherein the first data feature is {Ftext 1 , Ftext 2 , Ftext 3}, the second data feature is {Fspeech 1 , Fspeech 2 , Fspeech 3 , Fspeech 4}, data feature Ftext 1 The characteristic information is {Q 11 , K 11 , V 11}, data feature Ftext 2 The characteristic information is {Q 12 , K 12 , V 12}, data feature Ftext 3 The characteristic information is {Q 13 , K 13 , V 13}, data feature Fspeech 1 The characteristic information is {Q 21 , Q 21 , V 21}, data feature Fspeech 2 The characteristic information is {Q 22 , K 22 , V 22}, data feature Fspeech 3 The characteristic information is {Q 23 , K 23 , V 23}, data feature Fspeech 4 The characteristic information is {Q 24 , K 24 , V 24}, therefore, for the first feature representation S00, the characterization is the data feature Ftext 1The query vector and the data feature Fspeech 1 The key vector and value vector of 11 *K 21 T*V 21 , KT represents the transpose of K. At the same time, the data feature Fspeech 1 The query vector and the data feature Ftext 1 The key vector and value vector of 21 *K 11 T*V 11 , and finally Q 11 *K 21 T*V 21 and Q 21 *K 11 T*V 11 Add and fuse to obtain the first feature representation S00. Therefore, the first feature representation S00 can be expressed as Q 11 *K 21 T*V 21 +Q 21 *K 11 T*V 11 Similarly, we can obtain the first feature representation S01, the first feature representation S02, the first feature representation S03, the first feature representation S10, the first feature representation S11, the first feature representation S12, the first feature representation S13, the first feature representation S14, the first feature representation S21, the first feature representation S22 and the first feature representation S23.
[0066] Optionally, the terminal device fuses the query vector of the first data feature with the key vector of each second data feature in multiple second data features through an interactivity mechanism to obtain a third fused vector. At the same time, the terminal device fuses the key vector of the first data feature with the query vector of each second data feature in multiple second data features through an interactivity mechanism to obtain a fourth fused vector. Finally, the third fused vector is fuses with the fourth fused vector to obtain a second feature representation.
[0067] Continuing with the above embodiment, for the second feature representation T00, the data feature Ftext 1 The query vector and data features Fspeech 1 The key vectors are merged to obtain Q 11 *K 21 T, KT represents the transpose of K. At the same time, the data feature Fspeech 1 The query vector and data features Ftext 1 The key vectors are merged to obtain Q 21 *K 11 T, and finally Q11 *K 21 T and Q 21 *K 11 T is added and fused to obtain the second feature representation T00. Therefore, the second feature representation T00 can be expressed as T00 = Q 11 *K 21 T+Q 21 *K 11 T. Similarly, we can get the second feature representation T01, the second feature representation T02, the second feature representation T03, the second feature representation T10, the second feature representation T11, the second feature representation T12, the second feature representation T13, the second feature representation T20, the second feature representation T21, the second feature representation T22 and the second feature representation T23.
[0068] Step 30: Fusing the first feature representation and the second feature representation to obtain a fused feature representation.
[0069] Optionally, since the second feature representation does not have the property of attention, the second feature representation needs to be normalized to obtain a normalized feature representation. Further, the terminal device fuses the first feature representation and the normalized feature representation to obtain a fused feature representation, as described in steps 301 to 304.
[0070] The embodiment of the present application captures the importance of each data feature during feature interaction through the first feature representation, and captures the correlation between each data feature through the second feature representation, so as to determine the correlation, mutual influence and dependency between each data feature. Therefore, after the first feature representation is fused and added to the second feature representation, the final fused feature representation obtained not only considers the correlation between the data features, but also considers the importance of the data features during feature interaction, so that the final fused feature representation more accurately and comprehensively describes the relationship between each data feature, and improves the richness, expression ability and expressiveness of the final fused feature representation. Therefore, it can more accurately describe the interactivity between each data feature, thereby improving the feature representation ability of the final fused feature representation.
[0071] In one embodiment, the description of steps 301 to 304 is as follows:
[0072] Step 301: construct a first feature matrix based on a plurality of first feature representations, and construct a second feature matrix based on a plurality of second feature representations.
[0073] Optionally, there are multiple first feature representations and multiple second feature representations, so the terminal device determines the number of rows and columns of the first feature matrix according to the dimension information of the data features included in the feature set, where the dimension information can be understood as the number of features. Further, the terminal device fills each first feature representation to the corresponding row and column position according to the position information of each first feature representation in the first feature matrix to generate the first feature matrix, as described in steps 3011 to 3013.
[0074] Similarly, the terminal device determines the number of rows and columns of the second feature matrix according to the dimension information of the data features included in the feature set, and fills each second feature representation to the corresponding row and column position according to the position information of each second feature representation to generate the second feature matrix.
[0075] In one embodiment, the description of steps 3011 to 3013 is as follows:
[0076] Step 3011, determining the number of rows and columns of the first feature matrix based on the dimension information of the data features included in the feature set;
[0077] Step 3012, determining the position information of each first feature representation in the first feature matrix;
[0078] Step 3013: Generate a first feature matrix based on each first feature representation, the position information of each first feature representation in the first feature matrix, and the number of rows and columns of the first feature matrix.
[0079] Optionally, the terminal device obtains dimension information of the data features included in the feature set. In one embodiment, the feature set includes a first data feature and a second data feature, and therefore, the dimension information of the data features included in the feature set can be understood as first dimension information of the first data feature and second dimension information of the second data feature, wherein the dimension information can be understood as the number of features.
[0080] In one embodiment, the first data feature is {Ftext 1 , Ftext 2 , Ftext 3}, the second data feature is {Fspeech 1 , Fspeech 2 , Fspeech 3 , Fspeech 4}, therefore, the first dimension information of the first data feature is 3, and the second dimension information of the second data feature is 4.
[0081] Furthermore, the terminal device determines the number of rows and columns of the first feature matrix based on the dimensional information of the data features included in the feature set. In one embodiment, if the first dimensional information is 3 and the second dimensional information is 4, the number of rows and columns of the first feature matrix is 4 rows and 3 columns or 3 rows and 4 columns.
[0082] Further, the terminal device determines the position information of each first feature representation in the first feature matrix. In one embodiment, the feature set includes the first data feature and the second data feature. The first feature representation Sij can be understood as the first feature representation obtained by fusing the i-th first data feature and the j-th second data feature. Therefore, the position information of the first feature representation Sij is the i-th row and j-th column, or the j-th column and i-th row in the first feature matrix. In one embodiment, the first feature representation is S01, indicating that the first feature representation S01 is obtained by fusing the first first data feature and the second second data feature. Therefore, the position information of the first feature representation S01 in the first feature matrix is determined to be the first row and the second column.
[0083] Furthermore, the terminal device obtains the first feature matrix according to each first feature representation, the position information of each first feature representation in the first feature matrix, and the number of rows and columns of the first feature matrix.
[0084] In one embodiment, it is determined that the number of rows and columns of the first feature matrix is 4 rows and 3 columns. Therefore, the first feature representation S00 is filled into the 1st row and 1st column of the first feature matrix, the first feature representation S01 is filled into the 1st row and 2nd column of the first feature matrix, the first feature representation S02 is filled into the 1st row and 3rd column of the first feature matrix, the first feature representation S10 is filled into the 2nd row and 1st column of the first feature matrix, the first feature representation S11 is filled into the 2nd row and 2nd column of the first feature matrix, and the first feature representation S12 is filled into the 1st row and 3rd column of the first feature matrix. The first characteristic matrix S_4*3 is obtained by filling the first characteristic representation S20 into the third row and first column of the first characteristic matrix, filling the first characteristic representation S21 into the third row and second column of the first characteristic matrix, filling the first characteristic representation S22 into the third row and third column of the first characteristic matrix, filling the first characteristic representation S30 into the fourth row and first column of the first characteristic matrix, filling the first characteristic representation S31 into the fourth row and second column of the first characteristic matrix, and filling the first characteristic representation S32 into the fourth row and third column of the first characteristic matrix. Figure 6 , Figure 6 It is a schematic diagram of the first characteristic matrix provided in an embodiment of the present application.
[0085] Similarly, the terminal device determines the number of rows and columns of the second feature matrix based on the dimension information of the data features included in the feature set. Further, the terminal device determines the position information of each second feature representation in the second feature matrix. In one embodiment, the feature set includes the first data feature and the second data feature. The second feature representation Tnm can be understood as the second feature representation obtained by fusing the nth first data feature and the mth second data feature. Therefore, the position information of the second feature representation Tnm is the nth row and the mth column, or the mth column and the nth row in the second feature matrix.
[0086] Furthermore, the terminal device obtains the second feature matrix according to each second feature representation, the position information of each second feature representation in the second feature matrix, and the number of rows and columns of the second feature matrix.
[0087] In one embodiment, the number of rows and columns of the second feature matrix is determined to be 4 rows and 3 columns. Therefore, the second feature representation T00 is filled into the 1st row and 1st column of the second feature matrix, the second feature representation T01 is filled into the 1st row and 2nd column of the first feature matrix, the second feature representation T02 is filled into the 1st row and 3rd column of the second feature matrix, the second feature representation T10 is filled into the 2nd row and 1st column of the second feature matrix, the second feature representation T11 is filled into the 2nd row and 2nd column of the second feature matrix, and the second feature representation T12 is filled into the 2nd row and 2nd column of the second feature matrix. The second feature matrix is filled with the second feature representation T20 to the third row and first column of the second feature matrix, the second feature representation T21 to the third row and second column of the second feature matrix, the second feature representation T22 to the third row and third column of the second feature matrix, the second feature representation T30 to the fourth row and first column of the second feature matrix, the second feature representation T31 to the fourth row and second column of the second feature matrix, and the second feature representation T32 to the fourth row and third column of the second feature matrix, to obtain the second feature matrix T_4*3, refer to Figure 7 , Figure 7 It is a schematic diagram of the second characteristic matrix provided in an embodiment of the present application.
[0088] Step 302: normalize the second characteristic matrix to obtain a third characteristic matrix.
[0089] Optionally, the terminal device normalizes the second feature representation in the second feature matrix to obtain a third feature matrix, as described in steps 3021 to 3023.
[0090] In one embodiment, the description of steps 3021 to 3023 is as follows:
[0091] Step 3021, determining a fourth feature representation based on the second feature representation;
[0092] Step 3022, normalizing the second feature representation based on the fourth feature representation to obtain a normalized feature representation of the second feature representation;
[0093] Step 3023: Determine a third feature matrix based on the normalized feature representation.
[0094] Optionally, the terminal device adds and fuses the second feature representations in the second feature matrix to obtain a fourth feature representation in the second feature matrix. Further, the terminal device calculates a quotient of the second feature representation and the fourth feature representation to obtain a normalized feature representation of the second feature representation.
[0095] In one embodiment, referring to Figure 7 , the second feature representation T00, the second feature representation T01 and the second feature representation T02 in the first row of the second feature matrix are added and fused to obtain the fourth feature representation T00+T01+T02 in the first row of the second feature matrix. Further, the second feature representation T00, the second feature representation T01 and the second feature representation T02 in the first row are respectively calculated with the fourth feature representation T00+T01+T02 to obtain the normalized feature representation T00 / (T00+T01+T02) of the second feature representation T00 in the first row, the normalized feature representation T01 / (T00+T01+T02) of the second feature representation T01, and the normalized feature representation T02 / (T00+T01+T02) of the second feature representation T02. Similarly, we can obtain the normalized feature representation T10 / (T10+T11+T12) of the second feature representation T10 in the second row, the normalized feature representation T11 / (T10+T11+T12) of the second feature representation T11 in the second row, the normalized feature representation T12 / (T10+T11+T12) of the second feature representation T12 in the second row, the normalized feature representation T20 / (T20+T21+T22) of the second feature representation T21 in the third row. The normalized feature representation of the second feature representation T21 / (T20+T21+T22) in the third row is T22 / (T20+T21+T22), the normalized feature representation of the second feature representation T30 in the fourth row is T30 / (T30+T31+T32), the normalized feature representation of the second feature representation T31 in the fourth row is T31 / (T30+T31+T32), and the normalized feature representation of the second feature representation T32 in the fourth row is T32 / (T30+T31+T32).
[0096] Further, the terminal device obtains a third feature matrix according to the normalized feature representation of the second feature representation, referring to Figure 8 , Figure 8 It is a schematic diagram of the third characteristic matrix provided in an embodiment of the present application.
[0097] Step 303: merge the first characteristic matrix and the third characteristic matrix to obtain a fourth characteristic matrix.
[0098] Optionally, the terminal device fuses the first feature representation in the first feature matrix with its corresponding normalized feature representation in the third feature matrix according to the position information of the first feature representation in the first feature matrix to obtain a third feature matrix, specifically:
[0099] Optionally, the terminal device obtains the first feature representation identical to the position information in the first feature matrix according to the position information of each normalized feature representation in the third feature matrix, and performs point multiplication fusion on each normalized feature representation and its corresponding first feature representation to obtain a third feature representation corresponding to each normalized feature representation. Figure 6 and Figure 8 , for the normalized feature representation T00 / (T00+T01+T02) of the 1st row and 1st column in the third feature matrix, obtain the first feature representation S00 of the 1st row and 1st column in the first feature matrix, perform dot product fusion on the normalized feature representation T00 / (T00+T01+T02) and the first feature representation S00, and obtain the third feature representation S00*T00 / (T00+T01+T02) of the 1st row and 1st column. Similarly, the third feature representation S01*T01 / (T00+T01+T02) of the 1st row and 2nd column, the third feature representation S02*T02 / (T00+T01+T02) of the 1st row and 3rd column, the third feature representation S10*T10 / (T10+T11+T12) of the 2nd row and 1st column, and the third feature representation S10*T10 / (T10+T11+T12) of the 2nd row and 2nd column can be obtained. The third feature in the second row and third column represents S11*T11 / (T10+T11+T12), the third feature in the third row and first column represents S12*T12 / (T10+T11+T12), the third feature in the third row and first column represents S20*T20 / (T20+T21+T22), the third feature in the third row and second column represents S21*T21 / (T20+T21+T22), the third feature in the third row and third column represents S22*T21 / (T20+T21+T22), The third characteristic representation of the 4th row and the 1st column is S22*T22 / (T20+T21+T22), the third characteristic representation of the 4th row and the 2nd column is S30*T30 / (T30+T31+T32), the third characteristic representation of the 4th row and the 2nd column is S31*T31 / (T30+T31+T32), and the third characteristic representation of the 4th row and the 3rd column is S32*T32 / (T30+T31+T32). Further, the terminal device constructs a fourth characteristic matrix according to the third characteristic representation, referring to Fig. 9 , Fig. 9 It is a schematic diagram of the fourth characteristic matrix provided in an embodiment of the present application.
[0100] Step 304: fuse the third feature representations in the fourth feature matrix to obtain a fused feature representation.
[0101] Optionally, the terminal device adds and fuses the third feature representations in the fourth feature matrix to obtain the fifth feature representation in the fourth feature matrix. Fig. 9 , the fifth feature of the first row in the fourth feature matrix is represented as S00*T00 / (T00+T01+T02)+S01*T01 / (T00+T01+T02)+S02*T02 / (T00+T01+T02). Similarly, the fifth feature of the second row in the fourth feature matrix is represented as S10*T10 / (T10+T11+T12)+S11*T11 / (T10+T11+T12)+S12*T12 / (T10+T11+T12). The fifth feature of the third row is represented as S 20*T20 / (T20+T21+T22)+S21*T21 / (T20+T21+T22)+S22*T22 / (T20+T21+T22) and the fifth feature representation of the fourth row S30*T30 / (T30+T31+T32)+S31*T31 / (T30+T31+T32)+S32*T32 / (T30+T31+T32). Further, the terminal device determines the fifth feature representation in the fourth feature matrix as a fused feature representation. The fused feature representation specifically refers to Fig.10 , Fig.10 It is a schematic diagram of the fusion feature representation provided in the embodiment of the present application.
[0102] The embodiment of the present application adds the first feature representation to the second feature representation by means of a feature matrix. Therefore, the final fused feature representation not only considers the correlation between the data features, but also considers the importance of the data features during feature interaction, so that the final fused feature representation more accurately and comprehensively describes the relationship between the various data features, thereby improving the richness, expression ability and expressiveness of the final fused feature representation. Therefore, it can more accurately describe the interactivity between the various data features, thereby improving the feature representation ability of the final fused feature representation.
[0103] It should be understood that, although each step in the flowcharts involved in the above-mentioned embodiments is displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0104] Based on the same inventive concept, the embodiment of the present application also provides a feature fusion device for implementing the feature fusion method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of one or more feature fusion devices and feature fusion devices provided below can refer to the above limitations on feature fusion methods and feature fusion methods, and the specific limitations will not be repeated here.
[0105] This embodiment also provides a feature fusion device, which can be integrated into a terminal device. Fig.11 As shown, the feature fusion device may include:
[0106] The feature acquisition module 1101 is used to acquire a feature set; the feature set includes at least two types of data features;
[0107] A first feature fusion module 1102 is used to fuse feature information of the data features included in the feature set to obtain a first feature representation and a second feature representation; the first feature representation represents the importance of the data features included in the feature set during feature interaction; the second feature representation represents the association relationship between the data features included in the feature set;
[0108] The second feature fusion module 1103 is used to fuse the first feature representation and the second feature representation to obtain a fused feature representation.
[0109] The above-mentioned feature fusion device captures the importance of each data feature during feature interaction through the first feature representation, and captures the correlation between each data feature through the second feature representation, so as to determine the correlation, mutual influence and dependency between each data feature. Therefore, after the first feature representation is fused and added to the second feature representation, the final fused feature representation obtained not only considers the correlation between the data features, but also considers the importance of the data features during feature interaction, so that the final fused feature representation more accurately and comprehensively describes the relationship between each data feature, and improves the richness, expression ability and expressiveness of the final fused feature representation. Therefore, it can more accurately describe the interactivity between each data feature, thereby improving the feature representation ability of the final fused feature representation.
[0110] In some embodiments, the first feature fusion module 1102 is further used to:
[0111] A query vector, a key vector, and a value vector of the data features included in the feature set are fused to obtain a fused vector; and the first feature representation and the second feature representation are determined based on the fused vector.
[0112] In some embodiments, the first feature fusion module 1102 is further used to:
[0113] Fusing the query vector of the first data feature with the key vectors and the value vectors of the plurality of second data features to obtain a first fused vector;
[0114] Fusing the key vector and the value vector of the first data feature with the query vectors of the plurality of second data features to obtain a second fused vector;
[0115] Fusing the query vector of the first data feature with the key vectors of the plurality of second data features to obtain a third fused vector;
[0116] Fusing the key vector of the first data feature with the query vectors of the plurality of second data features to obtain a fourth fused vector;
[0117] The first fused vector is fused with the second fused vector to obtain the first feature representation; and the third fused vector is fused with the fourth vector to obtain the second feature representation.
[0118] In some embodiments, the second feature fusion module 1103 is further used to:
[0119] constructing a first feature matrix based on a plurality of the first feature representations, and constructing a second feature matrix based on a plurality of the second feature representations;
[0120] Normalizing the second characteristic matrix to obtain a third characteristic matrix;
[0121] The first characteristic matrix and the third characteristic matrix are merged to obtain a fourth characteristic matrix;
[0122] The third feature representations in the fourth feature matrix are fused to obtain the fused feature representation.
[0123] In some embodiments, the second feature fusion module 1103 is further used to:
[0124] determining a fourth feature representation based on the second feature representation;
[0125] Normalizing the second feature representation based on the fourth feature representation to obtain a normalized feature representation of the second feature representation;
[0126] The third feature matrix is determined based on the normalized feature representation.
[0127] In some embodiments, the second feature fusion module 1103 is further used to:
[0128] Determining the number of rows and columns of the first feature matrix based on the dimension information of the data features included in the feature set;
[0129] Determine the position information of each first feature representation in the first feature matrix
[0130] The first feature matrix is generated based on each of the first feature representations, position information of each of the first feature representations in the first feature matrix, and the number of rows and columns of the first feature matrix.
[0131] Based on the same inventive concept, the embodiment of the present application also provides a computer device, which may be a server or a terminal device, and includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned feature fusion method when executing the computer program. Thereby, various functions are realized, such as:
[0132] Acquire a feature set; the feature set includes at least two types of data features;
[0133] Based on the feature information of the data features included in the feature set, a first feature representation and a second feature representation are obtained; the first feature representation represents the importance of the data features included in the feature set during feature interaction; the second feature representation represents the association relationship between the data features included in the feature set;
[0134] The first feature representation and the second feature representation are fused to obtain a fused feature representation.
[0135] Beneficial effect: By capturing the importance of each data feature during feature interaction through the first feature representation, and capturing the correlation between each data feature through the second feature representation, the correlation, mutual influence and dependency between each data feature can be determined. Therefore, after the first feature representation is fused and added to the second feature representation, the final fused feature representation obtained not only considers the correlation between the data features, but also considers the importance of the data features during feature interaction, so that the final fused feature representation more accurately and comprehensively describes the relationship between each data feature, and improves the richness, expression ability and expressiveness of the final fused feature representation. Therefore, it can more accurately describe the interactivity between each data feature, thereby improving the feature representation ability of the final fused feature representation.
[0136] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0137] In one embodiment, the computer device is a terminal device, for example, its internal structure diagram can be as follows: Fig.12 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a feature fusion method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.
[0138] Those skilled in the art will understand that Fig.12The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0139] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.
[0140] Since the computer program stored in the computer-readable storage medium can execute any one of the feature fusion methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the feature fusion methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0141] Based on the same inventive concept, the embodiment of the present application further provides a computer program product or a computer program, which includes a computer program stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the methods provided in various optional implementations in the above embodiments.
[0142] It should be noted that the object data (including but not limited to user device information, user personal information, etc.) and conversation data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by computer programs and computer program-related hardware, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.
[0143] Any reference to the memory, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Re RAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0144] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.
[0145] In the above-mentioned feature fusion device, computer-readable storage medium, computer equipment, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the feature fusion device, computer-readable storage medium, computer program product, computer equipment and its corresponding units described above can refer to the description of the feature fusion method in the above embodiment, and will not be repeated here.
[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0147] The above is a detailed introduction to a feature fusion method, device, computer equipment, computer-readable storage medium and computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A feature fusion method, It is characterized in that The method comprises: Acquire a feature set; the feature set includes at least two types of data features; Based on the feature information of the data features included in the feature set, a first feature representation and a second feature representation are obtained; the first feature representation represents the importance of the data features included in the feature set during feature interaction; the second feature representation represents the association relationship between the data features included in the feature set; The first feature representation and the second feature representation are fused to obtain a fused feature representation.
2. The feature fusion method according to claim 1, It is characterized in that The feature information of the data feature includes a query vector, a key vector and a value vector; The fusing feature information based on the data features included in the feature set to obtain a first feature representation and a second feature representation includes: A query vector, a key vector, and a value vector of the data features included in the feature set are fused to obtain a fused vector; and the first feature representation and the second feature representation are determined based on the fused vector.
3. The feature fusion method according to claim 2, It is characterized in that The feature set includes a first data feature and a plurality of second data features; The query vector, the key vector and the value vector based on the data features included in the feature set are fused to obtain a fused vector; And determining the first feature representation and the second feature representation based on the fusion vector, comprising: Fusing the query vector of the first data feature with the key vectors and the value vectors of the plurality of second data features to obtain a first fused vector; Fusing the key vector and the value vector of the first data feature with the query vectors of the plurality of second data features to obtain a second fused vector; Fusing the query vector of the first data feature with the key vectors of the plurality of second data features to obtain a third fused vector; Fusing the key vector of the first data feature with the query vectors of the plurality of second data features to obtain a fourth fused vector; The first fused vector is fused with the second fused vector to obtain the first feature representation; and the third fused vector is fused with the fourth vector to obtain the second feature representation.
4. The feature fusion method according to claim 1, It is characterized in that The first characteristic indicates that there are multiple, and the second characteristic indicates that there are multiple; The fusing the first feature representation and the second feature representation to obtain a fused feature representation includes: constructing a first feature matrix based on a plurality of the first feature representations, and constructing a second feature matrix based on a plurality of the second feature representations; Normalizing the second characteristic matrix to obtain a third characteristic matrix; The first characteristic matrix and the third characteristic matrix are merged to obtain a fourth characteristic matrix; The third feature representations in the fourth feature matrix are fused to obtain the fused feature representation.
5. The feature fusion method according to claim 4, It is characterized in that The step of normalizing the second characteristic matrix to obtain a third characteristic matrix includes: determining a fourth feature representation based on the second feature representation; Normalizing the second feature representation based on the fourth feature representation to obtain a normalized feature representation of the second feature representation; The third feature matrix is determined based on the normalized feature representation.
6. The feature fusion method according to claim 4, It is characterized in that The constructing a first feature matrix based on the plurality of first feature representations comprises: Determining the number of rows and columns of the first feature matrix based on the dimension information of the data features included in the feature set; Determine position information of each of the first feature representations in the first feature matrix; The first feature matrix is generated based on each of the first feature representations, position information of each of the first feature representations in the first feature matrix, and the number of rows and columns of the first feature matrix.
7. A feature fusion device, It is characterized in that The device comprises: A feature acquisition module, used to acquire a feature set; the feature set includes at least two types of data features; A first feature fusion module, configured to fuse feature information of the data features included in the feature set to obtain a first feature representation and a second feature representation; the first feature representation represents the importance of the data features included in the feature set during feature interaction; the second feature representation represents the association relationship between the data features included in the feature set; The second feature fusion module is used to fuse the first feature representation and the second feature representation to obtain a fused feature representation.
8. A computer device, It is characterized in that It comprises a processor and a memory, wherein the memory stores a plurality of computer programs; the processor loads the computer program from the memory to execute the feature fusion method according to any one of claims 1 to 6.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a plurality of computer programs, and the computer programs are suitable for being loaded by a processor to execute the feature fusion method according to any one of claims 1 to 6.
10. A computer software, It is characterized in that The computer software includes a computer program, and the computer program is executed by a processor to implement the feature fusion method according to any one of claims 1 to 6.