Text classification method based on multimodal deep learning, device, and storage medium

Abstract

Disclosed in the present invention is a text classification method based on multimodal deep learning, comprising: acquiring context information of text data and weight information of text semantic features; and combining the context information and the weight information of the text semantic features via a fully connected layer, and then outputting the combined information to a classifier to calculate a score to obtain classification result information. The method effectively improves the accuracy and generalization ability of classification of texts generated from audios and videos, thereby improving the user experience.

Description

Text classification method, device and storage medium based on multimodal deep learning Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a text classification method, device, and storage medium based on multimodal deep learning. Background Art

[0002] With the application of multimodal deep learning technology, most of the interactions between smart refrigerators and users are voice and text data. Not only are the interactions based on video data very rare, but traditional methods for refrigerator smart voice and video generally have the following problems: inaccurate and insufficient feature extraction, resulting in low voice recognition accuracy and text classification accuracy of video content, which affects the user experience of refrigerator audio and video, and even affects the intelligence and information level of high-end refrigerators.

[0003] Therefore, building a text classification model for refrigerator audio and video using a multi-channel, multi-scale deep convolutional neural network model has become a key technology for improving text classification accuracy. Smart refrigerator interactions rely on heterogeneous data from multiple sources, such as voice, text, and video. Therefore, the industry has yet to come up with an effective solution for extracting optimal feature information from this heterogeneous data using multimodal or cross-modal data to optimize the accuracy of text classification for smart refrigerator audio and video, thereby improving the user experience. Summary of the Invention

[0004] The purpose of the present invention is to provide a text classification method, device and storage medium based on multimodal deep learning.

[0005] The present invention provides a method for generating text classification based on multimodal deep learning, comprising the steps of:

[0006] Acquire real-time audio and video data and historical audio and video data; pre-process the real-time audio and video data and historical audio and video data to obtain valid voice data and video data; transcribe the valid voice data into voice text data; acquire a video image of a local area in the valid video data, and transcribe the video image into image text data; acquire context information of the text data and weight information of text semantic features based on the voice text data and image text data; combine the context information and weight information of text semantic features through a fully connected layer, and output the combined information to a classifier to calculate a score to obtain classification result information, and determine the category information of the text generated by the audio and video data; and output the category information of the generated text.

[0007] As a further improvement of the present invention, the "preprocessing of the real-time audio and video data and the historical audio and video data to obtain valid voice data and video data" specifically includes: performing data cleaning, format analysis, format conversion and data storage on the real-time audio and video data and the historical audio and video data to obtain valid audio and video data; using a script or a third-party tool to separate the voice and video of the valid audio and video data to obtain the voice data and video data; preprocessing the voice data and video data, including: framing and windowing the voice data, and cropping and framing the video data.

[0008] As a further improvement of the present invention, the "transcription of the valid voice data into voice text data" specifically includes: extracting the features of the valid voice data to obtain voice features; inputting the voice features into a voice recognition multi-channel and multi-size deep convolutional neural network model to transcribe the first voice text data; outputting the alignment relationship between the voice features and the first voice text data based on a connection time series classification method to obtain second voice text data; based on an attention mechanism, obtaining the key features of the second voice text data or the weight information of the key features; combining the second voice text data and its key features or the weight information of the key features through a fully connected layer, and then calculating the score through a classification function to obtain the voice text data.

[0009] As a further improvement of the present invention, the “extracting the effective voice data features” specifically includes: extracting the effective voice data features and obtaining the Mel-frequency cepstral coefficient features thereof.

[0010] As a further improvement of the present invention, the "obtaining a video image of a local area in the video data, and transcribing the video image into image text data" specifically includes: obtaining a video image of the lip area based on the valid video data; inputting the video image of the lip area into a 3D convolutional neural network model for calculation to obtain image features; based on an image lip reading recognition method, inputting the image features into a multi-channel multi-scale time depth convolutional neural network model for transcribing to obtain first image text data; based on a connection temporal classification method, outputting the alignment relationship between the speech feature sequence and the first image text data to obtain second image text data; combining the second image text data through a fully connected layer, and then calculating the score through a classification function to obtain the image text data.

[0011] As a further improvement of the present invention, the method of "inputting the video image of the lip area into a 3D convolutional neural network model for calculation to obtain image features" specifically includes: segmenting the local video data of the lips into continuous lip picture frames; inputting the continuous lip picture frames into a 3D convolutional neural network model for calculation, extracting multiple features, and obtaining the image features.

[0012] As a further improvement of the present invention, the "based on the image lip reading recognition method, inputting the image features into the multi-channel multi-scale time deep convolutional neural network model for transcription to obtain the first image text data" specifically includes: inputting the image features into the multi-channel multi-scale time deep convolutional neural network for calculation to obtain time series image features; according to the image lip reading recognition method, mapping the time series image features into a pinyin sequence of a pinyin sentence; and then translating the pinyin sequence into a Chinese character sequence corresponding to the Chinese character sentence.

[0013] As a further improvement of the present invention, the "obtaining context information of the text data and weight information of text semantic features based on the speech text data and image text data" specifically includes: converting the speech text data and image text data into speech text word vectors and image text word vectors; inputting the speech text word vectors and image text word vectors into a bidirectional long short-term memory network model to obtain a context feature vector containing feature information of the speech text data and image text data.

[0014] As a further improvement of the present invention, based on the attention mechanism model, the self-weight information and / or associated weight information of the words and phrases in the text features of the speech text data and the image text data are distinguished to obtain the weight information of the text semantic features.

[0015] As a further improvement of the present invention, the "distinguishing the self-weight information and / or associated weight information of words and phrases in the text features of the speech text data and the image text data based on the attention mechanism model" specifically includes: inputting the speech text context feature vector and the image text context feature vector into the self-attention mechanism and the mutual attention mechanism respectively; obtaining the self-weight text attention feature vector containing the self-weight information of the speech text semantic features and the image text semantic features; obtaining the associated weight text attention feature vector containing the associated weight information of the speech text semantic features and the image text semantic features.

[0016] As a further improvement of the present invention, the "combining the context information and weight information through a fully connected layer, outputting the results to a classifier to calculate the score to obtain classification result information, and judging the category information of the text generated by the audio and video data" specifically includes: combining the context feature vector and the text attention weight feature vector through a fully connected layer, outputting the results to a classification function, calculating the scores of the text semantics of the speech text data and the image text data and their normalized scores, and obtaining the category information of the generated text.

[0017] As a further improvement of the present invention, the "transcription of the voice data into voice text data" also includes: obtaining configuration data stored in an external cache, executing the multi-channel and multi-scale deep convolutional neural network model calculation on the voice data based on the configuration data, performing text transcription and extracting text features.

[0018] The present invention also provides an electrical device, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned text classification method based on multimodal deep learning when running the executable instructions stored in the memory.

[0019] The present invention also provides a refrigerator, comprising: a memory for storing executable instructions; and a processor for implementing the above-mentioned text classification method based on multimodal deep learning when running the executable instructions stored in the memory.

[0020] The present invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the above-mentioned text classification method based on multimodal deep learning.

[0021] The beneficial effects of the present invention are as follows: the method provided by the present invention completes the task of recognizing and classifying the acquired audio and video generated text. First, by combining the speech text data with the speaker's lip area video image feature recognition text content, the text semantic feature information is complementary, correlated, and the semantic feature is enhanced, thereby achieving the accuracy of the audio and video generated text classification task. Secondly, by comprehensively considering real-time audio and video data and historical audio and video data, the historical audio and video data is used as supplementary data to make up for the problem of less semantic information in the speech data text and video text data, effectively improving the accuracy of text classification. Finally, by constructing a multi-channel multi-scale deep convolutional neural network and a temporal deep convolutional neural network model, the accuracy of real-time audio and video generated text classification and recognition is improved. Specifically, by constructing a convolutional neural network model that integrates context information mechanism, self-attention mechanism and mutual attention mechanism, the rich semantic feature information of the text data is more fully mined. Therefore, the overall model fully utilizes real-time and historical audio and video data as well as context data, has excellent semantic representation ability, and has a high accuracy rate for audio and video generated text classification, effectively improving the accuracy and generalization ability of audio and video generated text classification, and enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a structural block diagram of a model involved in a text classification method based on multimodal deep learning in one embodiment of the present invention.

[0023] FIG2 is a schematic diagram of the steps of a text classification method based on multimodal deep learning in one embodiment of the present invention.

[0024] FIG3 is a schematic diagram of steps for obtaining real-time audio and video data and historical audio and video data in one embodiment of the present invention.

[0025] FIG4 is a schematic diagram of data preprocessing steps for the real-time audio and video data and the historical audio and video data in one embodiment of the present invention.

[0026] FIG5 is a schematic diagram of the steps of transcribing the valid voice data into voice text data in one embodiment of the present invention.

[0027] FIG6 is a schematic diagram of the steps of obtaining a local area video image in the video data and transcribing the video image into image text data in one embodiment of the present invention.

[0028] FIG7 is a schematic diagram of steps for obtaining context information and weight information of text data based on the voice text data and image text data in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by those skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0030] It should be noted that the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In addition, the terms "first," "second," etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0031] An embodiment of the present invention is a text classification method based on multimodal deep learning. Although this application provides the method operation steps described in the following embodiment or flowchart 1, the order of execution of steps in the method that are not logically causally related is not limited to the order provided in the embodiment of this application, based on routine or no creative effort.

[0032] As shown in FIG1 , a structural block diagram of a model involved in a text classification method based on multimodal deep learning provided by the present invention is shown in FIG2 , which is a schematic diagram of the steps of the text classification method based on multimodal deep learning, including:

[0033] S1: Obtain real-time audio and video data and historical audio and video data.

[0034] S2: Preprocess the real-time audio and video data and the historical audio and video data to obtain valid voice data and video data.

[0035] S3: Transcribing the valid voice data into voice text data.

[0036] S4: Acquire a video image of a local area in the valid video data, and transcribe the video image into image text data.

[0037] S5: Acquire context information of the text data and weight information of text semantic features according to the speech text data and the image text data.

[0038] S6: After the context information and weight information are combined through a fully connected layer, they are output to a classifier to calculate a score to obtain classification result information, and the category information of the text generated by the audio and video data is determined.

[0039] S7: Outputting category information of the generated text.

[0040] The method provided by the present invention enables intelligent electronic devices to implement real-time user interaction or message push functionality based on real-time audio and video data input from users. In this embodiment, the method is illustrated using a smart refrigerator as an example, combined with a pre-trained deep learning model. Based on the user's audio and video input, the smart refrigerator classifies the corresponding text content generated by the user's audio and video data and calculates the required text content classification result information based on the classification result information.

[0041] As shown in FIG3 , in step S1, it specifically includes:

[0042] S11: Acquire the real-time audio and video data collected by the collection device, and / or

[0043] Acquire the real-time audio and video data transmitted from the client terminal.

[0044] S12: Obtaining historical audio and video data stored internally, and / or

[0045] Get historical audio and video data from external storage, and / or

[0046] Obtain historical audio and video data transmitted by client terminals.

[0047] The real-time audio and video data described herein includes real-time voice data and real-time video data. The real-time voice refers to the interrogative or directive statements currently spoken by the user to the intelligent electronic device or to a client terminal device that is communicatively connected to the intelligent electronic device. Similarly, it can also be the voice information collected by the voice collection device. For example, in this embodiment, the user can ask questions such as "What vegetables are in the refrigerator today?" or "What beef ingredients are in the refrigerator today?", or the user can issue commands such as "Delete all ingredients." The real-time video data is a real-time video image obtained by real-time capture using the intelligent electronic device or the client terminal device that the intelligent electronic device is communicatively connected to. For example, in this real-time method, the user's facial image is captured using a video camera built into the intelligent refrigerator, and the lip area feature image is extracted from the facial image to identify the text content corresponding to the image, such as identifying the image text data of "What vegetables are in the refrigerator today."

[0048] The historical audio and video data referred to herein refers to the user's real-time audio and video data from previous usage. Furthermore, it may also include historical audio and video data input by the user. Specifically, in this embodiment, the historical audio and video data may include: audio and video data obtained from previous commands or questions issued by the user. The obtained audio and video data contains information related to the current real-time audio and video data. It may also be explanatory audio and video information issued by the user based on the items placed in the refrigerator during previous usage, such as "There is no milk in the refrigerator." The acquisition of historical audio and video data can be used as part of the dataset for pre-training and prediction models, effectively supplementing the single voice representation of the real-time audio and video data and enriching the semantic features.

[0049] As described in step S11, in this embodiment, a camera or other audio and video acquisition device installed in the smart refrigerator can capture real-time audio and video of the user. During use, when the user needs to interact with the smart refrigerator, they can simply speak to the smart refrigerator. Furthermore, real-time audio and video data transmitted from the user can also be acquired via a client terminal connected to the smart refrigerator via a wireless communication protocol. The client terminal is an electronic device with information transmission capabilities, such as a mobile phone, tablet computer, smart camera, smart watch, app, or Bluetooth-enabled smart electronic device. During use, the user can simply speak to the client terminal or use the refrigerator's built-in camera to capture the audio and video. The client terminal captures the audio and video and transmits it to the smart refrigerator via wireless communication methods such as Wi-Fi or Bluetooth. This enables multi-channel real-time audio and video acquisition, and is not limited to speaking directly to the smart refrigerator. When the user needs to interact, real-time audio and video can be emitted through any convenient channel, significantly improving user convenience. In other embodiments of the present invention, one or more of the above-mentioned methods for acquiring real-time audio and video data may also be adopted, or the real-time audio and video data may be acquired through other channels based on existing technologies. The present invention does not impose specific limitations on this.

[0050] As described in step S12, in this embodiment, the historical audio and video data stored in the internal memory of the smart refrigerator can be read. In addition, the historical audio and video data stored in the external storage device configured for the smart refrigerator can also be read. The external storage device is a mobile storage device such as a U disk, SD card, etc. The storage space of the smart refrigerator can be further expanded by setting an external storage device. In addition, the historical audio and video data stored in a client terminal such as a mobile phone, a tablet computer, or an application software server can be obtained. Implementing multi-channel historical audio and video data acquisition channels can greatly increase the amount of historical audio and video data, thereby improving the accuracy of subsequent voice recognition and video image recognition. In other embodiments of the present invention, one or any multiple of the above-mentioned historical audio and video data acquisition methods can also be used, or the historical audio and video data can also be obtained through other channels based on existing technologies. The present invention does not impose specific restrictions on this.

[0051] Furthermore, in this embodiment, the smart refrigerator is equipped with an external cache, and at least part of the historical audio and video data is stored in the external cache. As the usage time increases, the historical audio and video data increases. By storing part of the data in the external cache, the internal storage space of the smart refrigerator can be saved, and when performing neural network calculations, the audio and video data stored in the external cache can be directly read, which can improve the efficiency of the algorithm.

[0052] Specifically, in this embodiment, a Redis component is used as the external cache. The Redis component is a widely used distributed cache system with a key / value storage structure, which can be used as a database, cache, and message queue proxy. In other embodiments of the present invention, other external caches such as Memcached may also be used, and the present invention does not impose specific limitations on this.

[0053] In summary, in step S11 and step S12, real-time audio and video data and historical audio and video data can be flexibly obtained through multiple channels, which improves the user experience while ensuring the data volume and effectively improving the algorithm efficiency.

[0054] As shown in FIG4 , in step S2, it specifically includes the following steps:

[0055] S21: performing data cleaning on the real-time audio and video data and the historical audio and video data to obtain valid audio and video data.

[0056] S22: Separate the valid audio and video data into voice and video to obtain the voice data and video data.

[0057] S23: Pre-processing the voice data and video data, including: performing framing and windowing processing on the voice data, and cropping and framing processing on the video data.

[0058] In step S21, data cleaning of the real-time audio and video data and the historical audio and video data specifically includes:

[0059] A certain number of real-time audio and video data sets and historical audio and video data sets are obtained. For example, they can be imported into the data cleaning model in the form of files for processing. In order to prevent data import failure, data that does not meet the file import format is parsed and converted, and then irrelevant data and duplicate data in the data set are deleted, and outliers and missing value data are processed. Information irrelevant to classification is preliminarily screened out, and the audio and video data is cleaned. At the same time, the cleaned data is output and saved in a specified format, thereby obtaining valid audio and video data.

[0060] In step S22, a script or a third-party audio and video separation tool is used to separate the effective audio and video data into voice and video, thereby obtaining voice data and video data.

[0061] In an embodiment of the present invention, Python language can be used to write an audio and video separation script, or a third-party audio and video separation tool can be used to separate the input audio and video data to achieve voice and video separation and obtain classified voice and video data.

[0062] In step S23, the classified speech is segmented according to a specified time period or number of samples, and the speech is framed to obtain speech signal data. Then, through the action of the window function, the speech signal originally containing noise exhibits characteristics of signal enhancement and signal periodicity, completing the windowing process, which facilitates the subsequent better extraction of speech feature parameters. Exemplarily, step S23 also includes cropping the valid video data to generate multiple frames of images. Specifically, a script can be written to first load the video data and read the video information, then decode the video based on the video information, determine how many images are displayed per second in the video, and thus obtain single-frame image information. The single-frame image information includes the width and height of each frame of the image, and finally save the video as multiple images. Therefore, after the processing of step S23, valid speech data and image data can be obtained. In other embodiments of the present invention, other video framing methods such as third-party video cropping tools can also be used, and the present invention does not specifically limit this.

[0063] As shown in FIG5 , in step S3, it specifically includes:

[0064] S31: Extracting the effective voice data features to obtain voice features.

[0065] S32: Input the speech features into a speech recognition multi-channel multi-size deep convolutional neural network model to transcribe and obtain first speech text data.

[0066] S33: Outputting the alignment relationship between the speech feature and the first speech text data based on the connection time series classification method to obtain second speech text data.

[0067] S34: Based on the attention mechanism, obtain the key features of the second speech text data or the weight information of the key features.

[0068] S35: The second speech text data and its key features or weight information of the key features are combined through a fully connected layer, and then a classification function is used to calculate a score to obtain the speech text data.

[0069] In step S31, extracting the effective voice data features specifically includes:

[0070] Extract the speech data features and obtain its Mel-scale Frequency Cepstral Coefficients (MFCCs). MFCCs are discernible components of speech signals and are cepstral parameters extracted in the Mel-scale frequency domain. The Mel scale describes the nonlinear characteristics of the human ear's frequency response. MFCC parameters take into account the human ear's sensitivity to different frequencies and are particularly suitable for speech recognition and speaker identification.

[0071] In an embodiment of the present invention, feature parameters such as perceptual linear prediction features (PLP) or linear prediction coefficient features (LPC) of the speech data may be obtained through different algorithm steps to replace the MFCC features. Specific adjustments may be made according to the actual application scenario and the model parameters used, and the present invention does not impose any specific restrictions on this.

[0072] The specific algorithm steps involved in the above steps can be referred to the current existing technology in this field, and the specific content will not be described in detail here.

[0073] In step S32, the effective voice data is transcribed into text content through a network model in automatic speech recognition technology to obtain the first voice text data.

[0074] In this embodiment, a multi-channel and multi-scale deep convolutional neural network model is constructed by increasing the width of the network to achieve the task of converting speech to text. The deep network model is composed of a multi-layer deep convolutional network model. The deep convolutional neural network model is generally composed of several convolutional layers and several fully connected layers, and contains various nonlinear operations and pooling operations in the middle. It is mainly used to process grid-structured data. Therefore, the model can use filters to filter out the contours between adjacent pixels. In addition, the model first proposes speech feature values ​​and then calculates the feature values ​​instead of calculating the original speech data values. Therefore, compared with traditional recurrent neural networks, deep convolutional neural network models have the advantages of low computational complexity and easy to characterize local features. In addition, shared weights and pooling layers can give the model better invariance in the time domain or frequency domain. In addition, the deeper nonlinear structure can also give the model powerful representation capabilities. In addition, multi-channel and multi-scale can extract speech features from different perspectives, obtain more speech feature information, and have better speech recognition accuracy.

[0075] Specifically, in this embodiment, in step S32, the multi-channel multi-size deep convolutional neural network used is composed of a 3*3 convolutional layer, 32 channels and a layer of maximum pooling.

[0076] In step S33 , the connectionist temporal classification (CTC) method is used to obtain the alignment relationship between the input speech feature sequence and the output speech text feature sequence.

[0077] In this embodiment, it is difficult to establish an accurate mapping relationship between the valid voice data and the text of the first voice text data, which increases the difficulty of subsequent voice recognition. To solve this problem, a time series classification method is adopted. This method is generally used after using a convolutional network model. It is a completely end-to-end acoustic model training that does not require pre-alignment of the data. Only one input sequence and one output sequence are required for training. There is no need to align the data and label them one by one. At the same time, the probability of sequence prediction can be directly output. Based on this predicted probability, we can obtain the most likely text output result to obtain the second voice text data.

[0078] Furthermore, in step S34, the attention mechanism can guide the deep convolutional neural network to focus on more critical feature information and suppress other non-critical feature information. Therefore, by introducing the attention mechanism, the local key features or weight information of the second speech text data can be obtained, thereby further reducing the irregular error alignment phenomenon of the sequence during model training.

[0079] Here, in step S35, based on the second speech text data and its key features or the weight information of the key features, the second speech text data is given its own weight information through a model that integrates the self-attention mechanism and the fully connected layer, so as to better obtain the internal weight information of the text semantic features of the speech text data to enhance the importance of different parts of the text semantic feature information, and finally the speech text data is obtained by calculating the score through the classification function.

[0080] As shown in FIG6 , in step S4, it specifically includes:

[0081] S41: Acquire a video image of the lip area according to the video data.

[0082] S42: Input the video image of the lip area into the 3D convolutional neural network model for calculation to obtain image features.

[0083] S43: Based on the image lip reading recognition method, the image features are input into a multi-channel multi-scale time deep convolutional neural network for transcription to obtain first image text data.

[0084] S44: Outputting the alignment relationship between the image features and the first image text data based on the connection temporal classification method to obtain second image text data.

[0085] S45: After combining the second image text data through the fully connected layer, the scores are calculated through the classification function to obtain the image text data.

[0086] In steps S41 and S42, considering that the sentences recognized may be complex based on the video image features of the human lip area, such as sentences of different lengths, different pause positions or word structures, and correlations between image features, we can perform video processing operations such as cropping and framing based on the valid video data to obtain a video image of the lip area, and crop and segment the video image of the lip area to obtain multiple continuous lip image frames. In this embodiment, the multiple continuous lip image frames are input into a 3D convolutional neural network model. By adding information in the time dimension, more expressive features can be extracted. The 3D convolutional neural network model can resolve the correlation information between multiple images. It uses multiple continuous frames of images as input and captures the motion information in the input frames by adding a new dimension of information, thereby better obtaining its image features.

[0087] In step S43, the results generated by the 3D convolutional neural network model in step S42 are input into a multi-channel, multi-scale, temporal, deep convolutional neural network model. After multi-channel, multi-convolution kernel operations are performed on the model, multiple feature maps are output, equal to the number of convolution kernels. For example, in a convolution layer with 3-channel input and 2 convolution kernels, 2 feature maps are output after the convolution calculation. Considering the sentence-level lip reading recognition method from video images, this embodiment utilizes two steps: pinyin sequence recognition (LipPic to Pinyin, P2P) and Chinese character sequence recognition (Pinyin to Chinese-Character, P2CC) to implement the video image lip reading recognition method. This method implements a Chinese lip reading recognition method. Specifically, the temporal image features generated by the multi-channel, multi-scale, temporal, deep convolutional neural network model are mapped to the pinyin sequence of a pinyin sentence, which is then translated into the Chinese character sequence of a Chinese sentence, ultimately obtaining the first image text data. Of course, there are no specific limitations on other Chinese lip reading recognition methods; as long as they can convert video images into corresponding text data, they are within the scope of protection of the present invention.

[0088] In steps S44 and S45, the continuous time series classification method is also used, which is the same as the above-mentioned voice data processing method, to realize the mapping relationship between the effective video data and the text of the first image text data, so as to obtain the second image text data. Then, the second image text data is given its own weight information and / or associated weight information through a model that integrates the self-attention mechanism and the fully connected layer, so as to better obtain the internal weight information and / or associated weight information of the text semantic features of the image text data, so as to enhance the importance of different parts of the text semantic feature information, and finally the image text data is obtained by calculating the score through the classification function. The specific processing process is the same as the above-mentioned voice data processing steps and will not be repeated here.

[0089] As shown in FIG7 , in step S5, it specifically includes:

[0090] S51: Convert the speech text data and image text data into speech text word vectors and image text word vectors.

[0091] S52: Input the speech text word vector and the image text word vector into a bidirectional long short-term memory network model to obtain a context feature vector containing feature information of the speech text data and the image text data.

[0092] S53: Based on the attention mechanism model, distinguish the self-weight information and / or associated weight information of the words and / or phrases in the text features of the speech text data and the image text data, and obtain the weight information of the text semantic features.

[0093] In step S51, in order to convert the text data into a vectorized form that can be recognized and processed by a computer, the speech text data and the image text data can be converted into the speech text word vector and the image text word vector through the Word2Vec algorithm, or the word vector can be obtained through other existing algorithms in the field such as the Glove algorithm, and the present invention does not impose any specific restrictions on this.

[0094] In step S52, a bidirectional long short-term memory (BiLSTM) network is composed of a forward LSTM network and a backward LSTM network. The LSTM model is better able to capture long-distance dependencies in text semantics, and based on this, the BiLSTM model is better able to capture bidirectional text semantics. The speech text word vector and image text word vector are input into the BiLSTM model. After being processed by the forward LSTM and backward LSTM, both the forward LSTM and the backward LSTM wait until all time steps are calculated before generating two result vectors. These two result vectors are then concatenated to output the context feature vector containing contextual information.

[0095] In the embodiment of the present invention, the voice data and video data may be transcribed into the voice text data and video text data by constructing a neural network model of other structures, and the specific method is not limited.

[0096] In step S53, in order to distinguish the weight information of different words or phrases in the speech text data and the image text data or the associated weight information between different text data, the speech text context feature vector and the image text context feature vector are respectively input into the self-attention mechanism and the mutual attention mechanism to obtain the self-weight feature vector containing the weight information of the speech text semantic features and the image text semantic features and the associated weight feature vector containing the associated weight information of the speech text semantic features and the image text semantic features, thereby making full use of the context information of audio and video to text conversion, supplementing the deficiency of single features in speech and video data, enriching the semantic representation ability in text data, and optimizing the subsequent text classification ability.

[0097] In step S6, it specifically includes:

[0098] The context feature vector and weighted text attention feature vector of the speech (including the self-weighted text attention feature vector and the associated weighted text attention feature vector) are combined through a fully connected layer and output to a classification function to calculate the scores of the text semantics in the speech text data and the image text data and their normalized score results to obtain classification result information.

[0099] In summary, the classification method for audio and video generated text provided by the present invention can be obtained by sequentially following the above steps. By obtaining the real-time audio and video data and historical audio and video data, performing data cleaning on them, and separating the voice and video, respectively, effective voice data and video data are generated, and both are used as part of the data set of the pre-training and prediction model, thereby more comprehensively obtaining the text semantic features. In addition, by constructing a multi-channel and multi-scale deep convolutional network model that integrates the connection time series classification method and the attention mechanism, as well as a video image lip reading recognition method based on the time deep convolutional neural network model and the sentence level, more abundant high-level semantic feature information is mined and obtained. Finally, by constructing a contextual information mechanism, a self-attention mechanism, and a mutual attention mechanism that integrates semantic text data and video text data, the semantic representation capability is more fully utilized, the shortcomings of single features in voice and video data are compensated, and the accuracy of audio and video generated text classification is improved. In addition, by obtaining externally stored configuration data for calculation, the computational efficiency of the model is improved. The overall model structure has a good semantic representation capability for text data, and the semantic feature entropy reflects good complementarity and correlation characteristics, thereby improving the accuracy of audio and video generated text classification.

[0100] In step S7, it specifically includes:

[0101] Convert the category information of the generated text into speech for output, and / or

[0102] Convert the category information of the generated text into voice and transmit it to the client terminal for output, and / or

[0103] Convert the category information of the generated text into text for output, and / or

[0104] Convert the category information of the generated text into text and transmit it to the client terminal for output, and / or

[0105] Convert the category information of the generated text into an image for output, and / or

[0106] The category information of the generated text is converted into an image and transmitted to the client terminal for output.

[0107] As described in step S7, in this real-time mode, after the classification result information is obtained through the above steps, it can be converted into voice and broadcasted through the built-in sound playback device of the smart refrigerator, or the result information can be converted into text and displayed directly through the display device configured in the smart refrigerator, or the result information can be converted into an image and displayed directly on the large screen of the smart refrigerator. In addition, the result information can also be transmitted by voice communication to the client terminal for output. Here, the client terminal is an electronic device with information receiving function, such as transmitting voice to a mobile phone, smart speaker, Bluetooth headset, etc. for broadcast, or the classification result information can be transmitted in text or image form through SMS, email, etc. to a client terminal such as a mobile phone, tablet computer, or an application software installed on the client terminal for user review. This realizes a multi-channel and multi-category classification result information output method. The user is not limited to obtaining relevant information only near the smart refrigerator. With the multi-channel and multi-category real-time voice acquisition method provided by the present invention, the user can directly interact with the smart refrigerator remotely, which is extremely convenient and greatly improves the user experience. In other embodiments of the present invention, only one or several of the above-mentioned classification result information output methods may be used, or the classification result information may be output through other channels based on existing technologies, and the present invention does not impose any specific restrictions on this.

[0108] In summary, the present invention provides a method for classifying audio and video generated text based on multimodal deep learning, which obtains real-time audio and video data and historical audio and video data through multiple channels, processes the audio and video data, and then converts the voice data and video data into corresponding voice text data and image text data. The context information after the audio and video text is generated is combined with the multi-channel multi-scale deep convolutional neural network model and the multi-channel multi-scale time deep convolutional neural network model to fully extract the text semantic features, obtain the generated text classification results, and output the text classification results through multiple channels. The method not only significantly improves the accuracy of generated text classification, but also makes the interaction between users and smart refrigerators more convenient and diversified, greatly improving the user experience.

[0109] Based on the same inventive concept, the present invention further provides an electrical device, comprising:

[0110] a memory for storing executable instructions;

[0111] The processor is configured to implement the above-mentioned text classification method based on multimodal deep learning when running the executable instructions stored in the memory.

[0112] Based on the same inventive concept, the present invention further provides a refrigerator comprising:

[0113] a memory for storing executable instructions;

[0114] The processor is configured to implement the above-mentioned text classification method based on multimodal deep learning when running the executable instructions stored in the memory.

[0115] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the above-mentioned text classification method based on multimodal deep learning.

[0116] It should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation method can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0117] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A text classification method based on multimodal deep learning, characterized in that: Includes steps: Obtain real-time audio and video data and historical audio and video data; Preprocessing the real-time audio and video data and the historical audio and video data to obtain valid voice data and video data; Transcribing the valid voice data into voice text data; Acquire a video image of a local area in the valid video data, and transcribe the video image into image text data; According to the speech text data and the image text data, obtaining context information of the text data and weight information of text semantic features; After combining the context information and the weight information through a fully connected layer, the context information and the weight information are output to a classifier to calculate a score to obtain classification result information, and to determine the category information of the text generated by the audio and video data; Output the category information of the generated text.

2. The text classification method based on multimodal deep learning according to claim 1, characterized in that: The "preprocessing the real-time audio and video data and the historical audio and video data to obtain valid voice data and video data" specifically includes: Performing data cleaning, format analysis, format conversion and data storage on the real-time audio and video data and historical audio and video data to obtain valid audio and video data; Using a script or a third-party tool to separate the effective audio and video data into voice and video to obtain the voice data and video data; The voice data and the video data are preprocessed, including: framing and windowing the voice data, and cropping and framing the video data.

3. The text classification method based on multimodal deep learning according to claim 1, characterized in that: The "transcribing the valid voice data into voice text data" specifically includes: Extracting the effective voice data features to obtain voice features; Inputting the speech feature into a speech recognition multi-channel multi-size deep convolutional neural network model to transcribe to obtain first speech text data; Outputting the alignment relationship between the speech feature and the first speech text data based on the connection time series classification method to obtain second speech text data; Based on the attention mechanism, obtaining key features of the second voice text data or weight information of the key features; The second speech text data and its key features or weight information of the key features are combined through a fully connected layer and then scored by a classification function to obtain the speech text data.

4. The text classification method based on multimodal deep learning according to claim 3 is characterized in that: The “extracting the effective voice data features” specifically includes: The effective speech data features are extracted to obtain the Mel-frequency cepstral coefficient features thereof.

5. The text classification method based on multimodal deep learning according to claim 1, characterized in that: The "obtaining a video image of a local area in the valid video data, and transcribing the video image into image text data" specifically includes: Acquire a video image of the lip area according to the video data; Inputting the video image of the lip area into a 3D convolutional neural network model for calculation to obtain image features; Based on the image lip reading recognition method, the image features are input into a multi-channel multi-scale time deep convolutional neural network model for transcription to obtain first image text data; Outputting the alignment relationship between the image feature and the first image text data based on the connection temporal classification method to obtain second image text data; The second image text data is combined through a fully connected layer and then scored by a classification function to obtain the image text data.

6. The text classification method based on multimodal deep learning according to claim 5, characterized in that: The “inputting the video image of the lip area into the 3D convolutional neural network model for calculation to obtain image features” specifically includes: Segmenting the lip local video data into continuous lip picture frames; The continuous lip picture frames are input into a 3D convolutional neural network model for calculation, and multiple features are extracted to obtain the image features.

7. The text classification method based on multimodal deep learning according to claim 6, characterized in that: The “based on the image lip reading recognition method, inputting the image features into a multi-channel multi-size time deep convolutional neural network model for transcription to obtain the first image text data” specifically includes: Input the image features into the multi-channel multi-size time deep convolutional neural network for calculation to obtain time series image features; According to the image lip reading recognition method, the time-series image features are mapped into a pinyin sequence of a pinyin sentence; The pinyin sequence is then translated into a Chinese character sequence corresponding to the Chinese character sentence.

8. The text classification method based on multimodal deep learning according to claim 1, characterized in that: The “obtaining context information of the text data and weight information of text semantic features according to the speech text data and the image text data” specifically includes: Converting the speech text data and the image text data into speech text word vectors and image text word vectors; The speech text word vector and the image text word vector are input into a bidirectional long short-term memory network model to obtain a context feature vector containing feature information of the speech text data and the image text data.

9. The text classification method based on multimodal deep learning according to claim 8, characterized in that: The method further comprises: Based on the attention mechanism model, the self-weight information and / or associated weight information of the words and phrases in the text features of the speech text data and the image text data are distinguished to obtain the weight information of the text semantic features.

10. The text classification method based on multimodal deep learning according to claim 9, characterized in that: The “distinguishing the self-weight information and / or associated weight information of words and phrases in the text features of the speech text data and the image text data based on the attention mechanism model” specifically includes: Inputting the speech text context feature vector and the image text context feature vector into a self-attention mechanism and a mutual attention mechanism respectively; Obtaining a self-weighted text attention feature vector including self-weight information of the speech text semantic features and the image text semantic features; Obtain an associated weight text attention feature vector containing associated weight information of the speech text semantic features and the image text semantic features.

11. The text classification method based on multimodal deep learning according to claim 10, characterized in that: The "combining the context information and the weight information through the fully connected layer, outputting the result information to the classifier to calculate the score to obtain the classification result information, and determining the category information of the text generated by the audio and video data" specifically includes: The context feature vector and the weighted text attention feature vector are combined through a fully connected layer and output to a classification function to calculate the text semantic scores of the speech text data and the image text data and their normalized score results to obtain the category information of the generated text.

12. The text classification method based on multimodal deep learning according to claim 1, characterized in that: The "transcribe the voice data into voice text data" also includes: The configuration data stored in the external cache is obtained, and the multi-channel and multi-size deep convolutional neural network model is calculated based on the configuration data to perform text transcription and extract text features.

13. An electrical device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the text classification method based on multimodal deep learning as described in any one of claims 1 to 12 when running the executable instructions stored in the memory.

14. A refrigerator, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the text classification method based on multimodal deep learning as described in any one of claims 1 to 12 when running the executable instructions stored in the memory.

15. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by the processor, the text classification method based on multimodal deep learning described in any one of claims 1 to 12 is implemented.

Images