Multi-source heterogeneous data feature extraction device, method, storage medium and electronic device

Abstract

The present invention discloses a multi-source heterogeneous data feature extraction device, method, storage medium and electronic device. The device includes a corresponding specific autoencoder, which is used to extract high-level features of each source data based on the low-level features of each source data; it includes a specific part encoder, a corresponding part encoder, a prediction decoder and a reconstruction decoder. The specific part encoder converts the low-level features of the corresponding source data into specific features, the corresponding part encoder converts the low-level features of the corresponding source data into corresponding features, the prediction encoder predicts the low-level features of other source data through the corresponding features, and the reconstruction decoder reconstructs the low-level features of the corresponding source data through the corresponding features and the corresponding features. By implementing the present invention, a corresponding specific autoencoder is set, and the disentangled corresponding features and specific features of the multi-source heterogeneous data are obtained through feature extraction. The problem that only corresponding features or specific features can be learned alone and the learned features are inaccurate is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-source heterogeneous data feature extraction device, method, storage medium and electronic equipment. Background Art

[0002] Currently, the decision-making of many tasks relies on information in various forms. For example, the diagnosis of the insulation performance of transformers in power systems requires comprehensive technical standards, test reports, papers and other materials, documents, pictures, videos and other information; whether the monitoring system alarms requires comprehensive information such as pedestrians, infrared, vibration, smoke concentration, and biometric fingerprints in the video; the diagnosis of cognitive impairment in the medical field requires comprehensive information such as daily behaviors and brain function imaging such as gait, sleep quality, and outdoor walking distance. The first task in building an effective retrieval and classification decision model is to explore feature extraction algorithms for multi-source heterogeneous data.

[0003] Usually, data from different sources contain both common parts and parts that are specific to each source. For example, Figure 1(a) shows dual-source heterogeneous data: a picture and its text description. Obviously, the concepts of "sunset" and "sea" can be seen in both the image and the text description, and are common parts of both image and text data. However, "Boracay", "iPhone8" and "good mood" in the text description cannot be directly obtained from the image, and are information specific to the text form. "People" and "sailboats" can only be seen in the image and cannot be obtained from the text. They are parts specific to the image form. Figure 1(b) uses a Venn diagram to give a set-based description of the above description. The circle on the left represents the set of image information, and the circle on the right represents the set of text information. The intersection of the two is the common part of the two source information.

[0004] Although there are many methods for extracting features from multi-source heterogeneous data in the prior art, most of the existing methods learn the unique part or the common part separately, and thus cannot clearly indicate that the corresponding features learned only contain the common part of the multi-source data, but exclude the specific part of each source data. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a multi-source heterogeneous data feature extraction device, method, storage medium and electronic device to solve the technical problem that the multi-source heterogeneous data feature extraction method in the prior art cannot accurately realize the joint extraction of common parts and specific parts.

[0006] The technical solution proposed by the present invention is as follows:

[0007] A first aspect of an embodiment of the present invention provides a multi-source heterogeneous data feature extraction device, including: corresponding specific autoencoders, the number of the corresponding specific autoencoders is the same as the number of data sources; the corresponding specific autoencoders are used to extract high-level features of each source data based on the low-level features of each source data, and the high-level features include specific features of each source data and corresponding features of multiple source data; the corresponding specific autoencoders include a specific part encoder, a corresponding part encoder, a prediction decoder and a reconstruction decoder, the specific part encoder converts the low-level features of the corresponding source data into specific features, the corresponding part encoder converts the low-level features of the corresponding source data into corresponding features, the prediction encoder predicts the low-level features of other source data through the corresponding features, and the reconstruction decoder reconstructs the low-level features of the corresponding source data through the corresponding features and the specific features.

[0008] Optionally, the corresponding specific autoencoder includes: an input layer, a corresponding feature layer, a specific feature layer, a prediction output layer and a reconstruction output layer, and there are correlation constraints between the corresponding feature layers of each corresponding specific autoencoder.

[0009] Optionally, the configuration parameters of the corresponding specific autoencoder include: connection weights and biases from the input layer to the corresponding feature layer, connection weights and biases from the input layer to the specific feature layer, connection weights and biases from the corresponding feature layer to the prediction output layer, connection weights and biases from the specific feature layer to the reconstructed output layer, and the connection weights from the corresponding feature layer to the prediction output layer in each corresponding specific autoencoder are consistent.

[0010] Optionally, the corresponding specific autoencoder is trained using a gradient descent algorithm; and configuration parameters of the corresponding specific autoencoder are fine-tuned using a gradient descent algorithm.

[0011] Optionally, the loss function of the corresponding specific autoencoder is determined based on the binary norm mapping between each source input layer feature, predicted output layer feature and reconstructed output layer feature, and the binary norm mapping between corresponding partial encoders.

[0012] Optionally, the corresponding specific autoencoder further includes: a gate of a prediction decoder, and the reconstruction decoder and the gate of the prediction decoder reconstruct low-level features of corresponding source data through the corresponding features and the specific features.

[0013] A second aspect of an embodiment of the present invention provides a multi-source heterogeneous data feature extraction method, which is applied to the first aspect of the embodiment of the present invention and the multi-source heterogeneous data feature extraction device described in any one of the first aspects. The method includes: using a feature extraction algorithm to extract low-level features of each source data; based on the low-level features of each source data, using a corresponding specific autoencoder to extract high-level features of each source data.

[0014] Optionally, the multi-source heterogeneous data feature extraction method also includes: calculating the distance between different source data based on corresponding features in the high-level features of each source data; retrieving matching objects based on the distance and performing a multi-source heterogeneous data retrieval task; training a multi-source data fusion classification model based on specific features in the high-level features of each source data and corresponding features of multiple source data, and performing a multi-source heterogeneous data classification task.

[0015] A third aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multi-source heterogeneous data feature extraction method as described in the second aspect of the embodiment of the present invention and any one of the second aspects.

[0016] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the multi-source heterogeneous data feature extraction method as described in the second aspect of the embodiment of the present invention and any one of the second aspects by executing the computer instructions.

[0017] The technical solution provided by the present invention has the following effects:

[0018] The multi-source heterogeneous data feature extraction device, method, storage medium and electronic device provided by the embodiment of the present invention extracts the low-level features of each source data obtained by setting a corresponding specific autoencoder, and obtains the disentangled corresponding features and specific features of the multi-source heterogeneous data. Since the corresponding specific autoencoder is provided with a specific part encoder, a corresponding part encoder, a prediction decoder and a reconstruction decoder, the corresponding features are processed by these encoders and decoders, thereby realizing the simultaneous extraction of specific features and corresponding features; and, through the conversion, prediction and reconstruction, it is ensured that the corresponding features only contain the common parts of the multi-source data, and the specific features only contain the specific parts of the multi-source data, which solves the problem that the processing method of the prior art can only learn the corresponding features or specific features alone, and the learned features are inaccurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] FIG1( a ) is a schematic diagram of dual-source heterogeneous data including images and texts;

[0021] FIG1( b ) is a schematic diagram of a set-based description for dual-source heterogeneous data using a Venn diagram;

[0022] Figure 1(c) is a characteristic diagram of learning common part data;

[0023] Figure 2 is a structural block diagram of a multi-source heterogeneous data feature extraction device according to an embodiment of the present invention;

[0024] Figure 3 is a structural block diagram of a multi-source heterogeneous data feature extraction device according to another embodiment of the present invention;

[0025] Figure 4 is a flowchart of a method for extracting features from multi-source heterogeneous data according to an embodiment of the present invention;

[0026] Figure 5 is a flowchart of a method for extracting features from multi-source heterogeneous data according to another embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of the structure of a computer-readable storage medium provided according to an embodiment of the present invention;

[0028] Figure 7 is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] The terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] As described in the background technology, in view of the characteristics of the existence of unique parts and common parts in multi-source data, most of the current processing methods are to learn the unique parts or the common parts separately. For example, one processing method is to learn a shared feature for data from multiple sources, and fuse the data from multiple sources to obtain a shared feature layer, which can be called a shared feature strategy. The goal of the model using this strategy is to learn a feature layer that can correspond to the Venn diagram in Figure 1(b), that is, some dimensions of the feature layer only contain image-specific information, some dimensions only contain text-specific information, and another part of the dimensions only contain common part information. However, the model using this strategy does not specify which dimensions of the feature layer represent the common part and which dimensions represent the specific part. It just assumes that the feature layer has learned such features.

[0032] Another approach is to learn the corresponding features for each source data separately, but add similarity constraints in the feature space of data from different sources to establish corresponding associations between multi-source data. This can be called the corresponding feature strategy. The goal of the model using this strategy is to learn a feature layer that can correspond to that shown in Figure 1(c), that is, the feature layer only contains the common parts of the image and text data. The optimization goal of the model using this strategy is to minimize matching (matching means that the image source data and the text source data are matched, such as {(Figure 1, Text 1), ( Figure 2 , Text 2)}), while maximizing the distance between the multi-source data features of mismatches (mismatches refer to the mismatch between image source data and text source data, such as (Figure 1, Text 2)).

[0033] According to different methods of maximizing the distance between unmatched multi-source data features, these models can be divided into two categories: one is to first construct unmatched multi-source data pairs, and then explicitly maximize the distance between these data pairs in the feature layer in the optimization objective; the other is to directly use the optimization objective of the feature learning model to make similar inputs have a smaller distance in the feature space and dissimilar inputs have a larger distance in the feature space, and cooperate with the optimization objective of minimizing the distance between matched multi-source data features, thus achieving the purpose of making the unmatched multi-source data pairs farther apart in the feature space.

[0034] Although the above processing methods have achieved excellent performance in the fusion and alignment tasks of multi-source heterogeneous data, so far, it has not been possible to clearly show that the corresponding features learned only contain the common parts of multi-source data and exclude the specific parts of each source data. Moreover, the features extracted by a better learning multi-source heterogeneous feature extraction model should have the ability to retain and distinguish the common parts and the specific parts at the same time. Such features can not only better complete the alignment and retrieval tasks of multi-source heterogeneous data, but also better complete the multi-source classification decision tasks.

[0035] In view of this, an embodiment of the present invention provides a multi-source heterogeneous data feature extraction device. By setting a corresponding specific autoencoder, the features of each source data are disentangled into corresponding features and specific features. The corresponding features are suitable for mining the relationship between multi-source data and performing retrieval tasks, and the specific features are suitable for performing multi-source data classification tasks together with the corresponding features.

[0036] The embodiment of the present invention provides a multi-source heterogeneous data feature extraction device, such as Figure 2 As shown, the device includes: corresponding specific autoencoders, the number of which is the same as the number of data sources; the corresponding specific autoencoders are used to extract high-level features of each source data based on the low-level features of each source data, and the high-level features include specific features of each source data and corresponding features of multiple source data; the corresponding specific autoencoders include a specific part encoder, a corresponding part encoder, a prediction decoder and a reconstruction decoder, the specific part encoder converts the low-level features of the corresponding source data into specific features, the corresponding part encoder converts the low-level features of the corresponding source data into corresponding features, the prediction encoder predicts the low-level features of other source data through the corresponding features, and the reconstruction decoder reconstructs the low-level features of the corresponding source data through the corresponding features and the specific features.

[0037] The multi-source heterogeneous data feature extraction device provided by the embodiment of the present invention extracts the low-level features of each source data obtained by setting a corresponding specific autoencoder, and obtains the disentangled corresponding features and specific features of the multi-source heterogeneous data. Since the corresponding specific autoencoder is provided with a specific part encoder, a corresponding part encoder, a prediction decoder and a reconstruction decoder, the corresponding features are processed by these encoders and decoders, thereby realizing the simultaneous extraction of specific features and corresponding features; and, through the conversion, prediction and reconstruction, it is ensured that the corresponding features only contain the common parts of the multi-source data, and the specific features only contain the specific parts of the multi-source data, which solves the problem that the processing method of the prior art can only learn the corresponding features or specific features alone, and the learned features are inaccurate.

[0038] In one embodiment, low-level features of each source data may be pre-extracted by feature extraction. The specific feature extraction method may be any existing method that can achieve feature extraction. Low-level features may be edge information, surface or apparent information in the data, such as texture features. High-level features are some higher-level semantic feature information.

[0039] Among them, for a specific data source, the low-level features of the source data are x (k) , then the specific part encoder is represented by f I→S , the corresponding part of the encoder is represented by f I→C, the predictive decoder is denoted as g C→I , the reconstruction decoder is represented as {g S→I ,g C→I}. Therefore, the corresponding feature is expressed as f I→C (x (k) ), the specific feature is represented by f I→S (x (k) ), the feature representation obtained by reconstructing the low-level features of the corresponding source data through the corresponding features and the specific features is If the low-level features of other source data are represented by y (k) , the corresponding encoder in other sources is denoted as f T→C , then the feature representation obtained by predicting the low-level features of other source data is

[0040] The Coordinated and Specific auto Encoder (COSE) has the ability to extract both corresponding features and specific features. The corresponding features correspond to the common part of data from different sources, through which the mutual prediction of data from different sources can be achieved. That is, the corresponding specific auto encoder corresponding to each source can predict the data from other sources through the corresponding features. For example, if the input data x (k) Through the encoder f I→C Get the corresponding features, and then pass the decoder g C→I To predict the data y from the second source (k) ; Similarly, the corresponding specific autoencoders from other sources pass their input data y (k) To predict the data x from this source (k) If this mutual prediction part is missing, only specific features will be involved in reconstructing the input, while the corresponding features will not be involved in reconstructing the input, that is, the corresponding features of the two sources are the same fixed value, the associated error is 0, and the corresponding specific autoencoders of the two sources each reconstruct the data of their own source, which is equivalent to two independent autoencoders. At this time, the overall error is the smallest. Obviously, the corresponding features of the corresponding specific autoencoders without the mutual prediction part do not contain any information, and the specific features contain all the information of each source data. Therefore, this part is indispensable for COSE.

[0041] Secondly, the specific features correspond to the individual parts of the corresponding source data. Although the individual parts of the data cannot be directly obtained as the reconstruction target, COSE uses the specific features and corresponding features to reconstruct the information of a single source. In this way, since a corresponding specific autoencoder can predict the data of another source through the corresponding features, when it reconstructs the data of the current source together with the specific features, its role is to reconstruct the common part of the multi-source data, so the specific features can focus on reconstructing the remaining part, that is, the individual parts of the corresponding source data. For example, in x (k) When reconstructing, the input source includes two parts: one is the specific features after the mapping function g S→I The decoding result of the corresponding feature is obtained by mapping function g C→I The result of decoding. Therefore, as long as the function g C→I The result of decoding is the common part of the two source data, which can make the function g S→I The decoding result is the input data x (k) Only in this way can the reconstruction of the input data x be completed. (k) After the model is trained, the corresponding features of the two aligned source data are close, and it undertakes the task of predicting the two source data. In other words, the mapping function g C→I The object of decoding is the corresponding features of the two source data, so the decoding result will tend to restore the common parts of the two source data.

[0042] In one embodiment, the corresponding specific autoencoder includes: an input layer, a corresponding feature layer, a specific feature layer, a prediction output layer, and a reconstruction output layer, and there are correlation constraints between the corresponding feature layers of each corresponding specific autoencoder. The configuration parameters of the corresponding specific autoencoder include: connection weights and biases from the input layer to the corresponding feature layer, connection weights and biases from the input layer to the specific feature layer, connection weights and biases from the corresponding feature layer to the prediction output layer, connection weights and biases from the specific feature layer to the reconstruction output layer, and the connection weights from the corresponding feature layer to the prediction output layer in each corresponding specific autoencoder are consistent.

[0043] Specifically, in order to make the corresponding features of the aligned multi-source heterogeneous data similar, a solution of explicitly adding the corresponding feature similarity constraint in the optimization objective can be adopted, or a solution of explicitly adding the corresponding feature similarity constraint in the optimization objective while also sharing the decoder weights of the corresponding features can be adopted. Because the corresponding feature layers of the data from the two sources are similar, the decoded reconstructed data x (k) and (k) The weights should also be consistent, that is, the connection weights from the corresponding feature layer to the prediction output layer in each corresponding specific autoencoder are consistent.

[0044] In one embodiment, the corresponding specific autoencoder is trained using a gradient descent algorithm; the configuration parameters of the corresponding specific autoencoder are fine-tuned using a gradient descent algorithm. Specifically, the structure of the specific part encoder, the corresponding part encoder, the prediction decoder, and the reconstruction decoder in the corresponding specific autoencoder can adopt the structure of an existing encoder or decoder, for example, it can be a neural network including a layer of hidden layer nodes, and the input and output have the same number of nodes. Corresponding to the above structure, the corresponding feature layer and the specific feature layer are the hidden layers in the corresponding specific autoencoder.

[0045] In one embodiment, the loss function of the corresponding specific autoencoder is determined according to the two-norm mapping between each source input layer feature, the predicted output layer feature and the reconstructed output layer feature and the two-norm mapping between the corresponding partial encoders. The corresponding specific autoencoder also includes: a gate of a prediction decoder, and the gates of the reconstruction decoder and the prediction decoder reconstruct the low-level features of the corresponding source data through the corresponding features and the specific features.

[0046] For the prediction decoder in the corresponding specific autoencoder, its input information only contains information that can decode the common part of the output, so the decoding result will contain noise of the prediction specific part. Since the connection weights from the corresponding feature layer to the prediction output layer are consistent, the noise will also affect the reconstruction of its own part in another corresponding specific autoencoder. Therefore, when reconstructing its own source data, a gate structure is added to filter out this part of the noise.

[0047] In one embodiment, it is assumed that N pieces of data constitute a multi-source heterogeneous data set, and the multi-source heterogeneous data set is represented by D = {z1, z2, ..., z k ,…,z N}, where z k ={x (k) ,y (k)} is the kth data, x (k) ,y (k) are low-level features of the first source data and the second source data of the data. Figure 3 As shown, by the corresponding specific autoencoder in the first source (including the specific partial encoder represented by f I→s , the corresponding part of the encoder is represented by f I→C , the predictive decoder is denoted as g C→I , the reconstruction decoder is represented as {g S→I ,g C→I}) and the corresponding specific autoencoder of the second source (including the specific partial encoder denoted as f T→S , the corresponding part of the encoder is represented by f T→C , the predictive decoder is denoted as g C→T , the reconstruction decoder is represented as {gS→T ,g C→T}) to predict and reconstruct, the first source corresponds to the specific autoencoder reconstructed output and the predicted output The second source corresponds to the specific autoencoder prediction output and reconstruct the output The specific expression is:

[0048]

[0049]

[0050]

[0051]

[0052] The configuration parameters of the corresponding specific autoencoder in the first source include: the connection weight W from the first source input layer to the common part of the hidden layer (corresponding to the feature layer) I→C and bias b I→C , the first source input layer to the hidden layer specific part (specific feature layer) connection weight W I→S and bias b I→S , the first source hidden layer public part to the predicted second source output layer connection weight W I→T and bias b I→T , the connection weight W from the first source hidden layer to the reconstructed first source output layer I→I and bias b I→I The configuration parameters of the corresponding specific autoencoder in the first source include: the connection weight W of the common part from the input layer to the hidden layer of the second source T→C and bias b T→C , the second source input layer to the hidden layer specific part connection weight W T→S and bias b T→S , the connection weight W of the common part of the second source hidden layer to the predicted first source output layer T→I and bias b T→I , the connection weight W from the second source hidden layer to the reconstructed second source output layer T→T and bias b T→T .

[0053] At the same time, based on the above content, the loss function corresponding to a specific autoencoder is expressed as:

[0054]

[0055] in,

[0056]

[0057]

[0058]

[0059] α represents a constant, α∈(0,1); is a two-norm mapping.

[0060] By training the corresponding specific autoencoder and fine-tuning the configuration parameters therein, training and adjustment are performed with the goal of minimizing the above-mentioned loss function, thereby obtaining the final configuration parameters.

[0061] When the gate structure of the prediction decoder is added to the corresponding specific autoencoder, the specific calculation method of the reconstructed corresponding source features is as follows:

[0062]

[0063]

[0064] Among them, σ C→I and σ C→T They are respectively the predictive decoder g C→I and g C→T door.

[0065] The embodiment of the present invention also provides a method for extracting features from multi-source heterogeneous data. Figure 4 As shown, the method is applied to the multi-source heterogeneous data feature extraction device described in the above embodiment, and the method includes the following steps:

[0066] Step S101: extract low-level features of each source data using a feature extraction algorithm. Specifically, the low-level features of each source data can be pre-extracted using a feature extraction method. The specific feature extraction method can be any existing method that can achieve feature extraction. The low-level features can be edge information, surface or apparent information in the data, such as texture features.

[0067] Step S102: Based on the low-level features of each source data, a corresponding specific autoencoder is used to extract high-level features of each source data. The structure of the corresponding specific autoencoder can adopt the structure in the above device embodiment, which will not be described in detail here.

[0068] The multi-source heterogeneous data feature extraction method provided by the embodiment of the present invention extracts the low-level features of each source data obtained by the corresponding specific autoencoder to obtain the disentangled corresponding features and specific features of the multi-source heterogeneous data. Since the corresponding specific autoencoder is provided with a specific part encoder, a corresponding part encoder, a prediction decoder and a reconstruction decoder, the corresponding features are processed by these encoders and decoders, thereby realizing the simultaneous extraction of specific features and corresponding features; and, through the conversion, prediction and reconstruction, it is ensured that the corresponding features only contain the common parts of the multi-source data, and the specific features only contain the specific parts of the multi-source data, which solves the problem that the processing method of the prior art can only learn the corresponding features or specific features alone, and the learned features are inaccurate.

[0069] In one embodiment, if Figure 5 As shown, the multi-source heterogeneous data feature extraction method also includes: calculating the distance between different source data based on corresponding features in the high-level features of each source data; retrieving matching objects based on the distance and performing a multi-source heterogeneous data retrieval task; training a multi-source data fusion classification model based on specific features in the high-level features of each source data and corresponding features of multiple source data, and performing a multi-source heterogeneous data classification task.

[0070] The retrieval task refers to the cross-source retrieval of multi-source heterogeneous data, that is, source 1 retrieves source 2 or source 2 retrieves source 1. Because the query and the candidate set (corresponding features) have corresponding representations after step 102, by calculating the corresponding distance between the query and the candidate set, the candidate set can be sorted according to the distance, with the closer distance being ranked first.

[0071] If two sources are involved, the classification task refers to combining all the features (corresponding features and specific features) of source 1 and source 2, training a classification model, and performing the classification task. The classification model can be any classification model, and the classification task can be any classification task related to sources 1 and 2. For example, taking the classification task of goods as an example, assuming that source 1 is the product image, source 2 is the product title, and the classification task is the category of the product. Then, the features of the two sources can be extracted using the method described in the invention, and then the classification task can be performed on all the features using any classification model.

[0072] For a detailed functional description of the multi-source heterogeneous data feature extraction method provided in an embodiment of the present invention, please refer to the description of the multi-source heterogeneous data feature extraction device in the above embodiment.

[0073] The embodiment of the present invention also provides a storage medium, such as Figure 6As shown, a computer program 601 is stored thereon, and when the instruction is executed by the processor, the steps of the multi-source heterogeneous data feature extraction method in the above embodiment are implemented. The storage medium also stores audio and video stream data, feature frame data, interaction request signaling, encrypted data, and preset data size, etc. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0074] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0075] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, the electronic device may include a processor 51 and a memory 52, wherein the processor 51 and the memory 52 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0076] The processor 51 may be a central processing unit (CPU). The processor 51 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0077] The memory 52 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the corresponding program instructions / modules in the embodiment of the present invention. The processor 51 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 52, that is, the multi-source heterogeneous data feature extraction method in the above method embodiment is implemented.

[0078] The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device and at least one function; the data storage area may store data created by the processor 51, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the processor 51 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0079] The one or more modules are stored in the memory 52 and when executed by the processor 51, perform the following steps: Figure 4 -5 is a multi-source heterogeneous data feature extraction method in the embodiment shown in FIG.

[0080] For details of the above electronic equipment, please refer to Figures 4 to 5 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0081] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multi-source heterogeneous data feature extraction device, characterized in that: include: Corresponding to specific autoencoders, the number of the corresponding specific autoencoders is the same as the number of data sources; The corresponding specific autoencoder is used to extract high-level features of each source data based on low-level features of each source data, wherein the high-level features include specific features of each source data and corresponding features of multiple source data, wherein the multiple source data include dual-source heterogeneous data, and the dual-source heterogeneous data include a set of image information and a set of text information; The corresponding specific autoencoder includes a specific part encoder, a corresponding part encoder, a prediction decoder and a reconstruction decoder. The specific part encoder converts the low-level features of the corresponding source data into specific features, the corresponding part encoder converts the low-level features of the corresponding source data into corresponding features, the prediction encoder predicts the low-level features of other source data through the corresponding features, and the reconstruction decoder reconstructs the low-level features of the corresponding source data through the corresponding features and the specific features.

2. The multi-source heterogeneous data feature extraction device according to claim 1, characterized in that: The corresponding specific autoencoder includes: an input layer, a corresponding feature layer, a specific feature layer, a prediction output layer and a reconstruction output layer, and there are correlation constraints between the corresponding feature layers of each corresponding specific autoencoder.

3. The multi-source heterogeneous data feature extraction device according to claim 2, characterized in that: The configuration parameters of the corresponding specific autoencoder include: connection weights and biases from the input layer to the corresponding feature layer, connection weights and biases from the input layer to the specific feature layer, connection weights and biases from the corresponding feature layer to the prediction output layer, connection weights and biases from the specific feature layer to the reconstructed output layer, and the connection weights from the corresponding feature layer to the prediction output layer in each corresponding specific autoencoder are consistent.

4. The multi-source heterogeneous data feature extraction device according to claim 1, characterized in that: The corresponding specific autoencoder is trained by using a gradient descent algorithm; The configuration parameters corresponding to the specific autoencoder are fine-tuned using a gradient descent algorithm.

5. The multi-source heterogeneous data feature extraction device according to claim 3, characterized in that: The loss function of the corresponding specific autoencoder is determined based on the binary norm mapping between each source input layer feature, predicted output layer feature and reconstructed output layer feature, and the binary norm mapping between the corresponding partial encoders.

6. The multi-source heterogeneous data feature extraction device according to claim 1, characterized in that: The corresponding specific autoencoder also includes: a gate of a prediction decoder, and the reconstruction decoder and the gate of the prediction decoder reconstruct low-level features of corresponding source data through the corresponding features and specific features.

7. A method for extracting features from multi-source heterogeneous data, characterized in that: The multi-source heterogeneous data feature extraction device according to any one of claims 1 to 6 comprises: Use feature extraction algorithms to extract low-level features from each source data; Based on the low-level features of each source data, the corresponding specific autoencoder is used to extract the high-level features of each source data.

8. The multi-source heterogeneous data feature extraction method according to claim 7, characterized in that: Also includes: Calculate the distances between different source data based on the corresponding features in the high-level features of each source data; Retrieve matching objects according to the distance and perform multi-source heterogeneous data retrieval tasks; According to the specific features in the high-level features of each source data and the corresponding features of multiple source data, a multi-source data fusion classification model is trained to perform multi-source heterogeneous data classification tasks.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multi-source heterogeneous data feature extraction method as described in claim 7 or 8.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-source heterogeneous data feature extraction method as described in claim 7 or 8 by executing the computer instructions.

Images