Method and device for extraction of user portrait labels for cold chain stowage based on multimodality

Abstract

The invention discloses a method and device for extracting labels of user portraits of cold chain stowage based on multimodality. The labeled text set is used as the input of the BERT model, and the long and short-term memory network is used to further extract features; YOLO is used to extract image features , the extracted bimodal features are supplemented with the correlation representation of the activation function tanh and other modal embeddings, the supplemented bimodal feature vectors are spliced, and they are matrix-multiplied with the bimodal condition vector, the result As the input of the Softmax function, the bimodal interaction attention matrix is ​​obtained; the bimodal interaction attention matrix is ​​spliced ​​with the supplemented bimodal features, and it is used as the input of the fully connected layer to obtain the intermodal interaction features and the internal modalities. Features, and finally input to Softmax for classification. The invention utilizes a multi-modal feature fusion algorithm to merge and fuse user features of different modalities, establishes an interaction relationship between different modalities, and reduces extraction noise.

Description

technical field [0001] The invention relates to the technical field of user portraits and multi-modal fusion, in particular to a method and device for extracting labels of user portraits for cold chain stowage based on multi-modalities. Background technique [0002] In recent years, user portraits have become a feature of research around the world, and are increasingly attracting widespread attention from the industry and academia. More importantly, it is also one of the key technologies for many applications. [0003] In the existing user portrait modeling technology, the research on how to integrate multiple data sources or modalities in order to obtain a more accurate user portrait is quite limited, and there are the following shortcomings: 1. Some user portrait research work only involves a single 2. A simple and easy integration method is to directly splicing multiple modal feature vectors when the information is input, but this method ignores the semantic differences o...

AI Technical Summary

Problems solved by technology

[0003] In the existing user portrait modeling technology, the research on how to integrate multiple data sources or modalities to obtain a more accurate user portrait is quite limited, and there are the following deficiencies: 1. Some user portrait research work only focuses on a single 2. The simple and easy integration method is to directly splice the feature vectors of various modalities when information is input, but this method ignores the semantic differences of different modal features. The interaction relationship between different modes has not been established, and there is a large noise interference

Images