Text classification method and apparatus for multi-modal data, and refrigeration device and medium

Abstract

The present application relates to the technical field of data processing. Disclosed is a text classification method for multi-modal data. The method comprises: obtaining text information, speech information and image information according to multi-modal data; obtaining a speech text feature vector and an image text feature vector according to the speech information and the image information; and performing text preprocessing on the text information, the speech text feature vector and the image text feature vector to obtain short-text data. Further disclosed in the present application are a text classification apparatus for multi-modal data, and a refrigeration device and a medium.

Description

Text classification method and device for multimodal data, refrigeration equipment, and medium

[0001] This application is based on the Chinese patent application with application number 202310694668.X and application date June 12, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0002] The present application relates to the field of data processing technology, for example, to a text classification method and device, refrigeration equipment, and medium for multimodal data. Background Art

[0003] With the rapid development of multimodal speech noise reduction, speech recognition, and signal processing technologies, extracting interactive speech or text from multimodal data has become a research hotspot in the industry. In smart home scenarios, intelligent cognition and perception suffer from slow interaction and high feedback error rates, impacting the user experience. Therefore, extracting high-quality short text from multimodal data has become a pressing technical challenge.

[0004] The related technology discloses a text classification method based on multimodal deep learning, including: S01, obtaining real-time audio and video data and historical audio and video data. S02, preprocessing the real-time audio and video data and historical audio and video data to obtain valid voice data and video data. S03, transcribing the valid voice data into voice text data. S04, obtaining the video image of the local area in the valid video data, and transcribing the video image into image text data. S05, obtaining the context information of the text data and the weight information of the text semantic features based on the voice text data and the image text data. S06, combining the context information and the weight information through the fully connected layer, outputting them to the classifier to calculate the score to obtain the classification result information, and judging the type information of the text generated by the audio and video data. S07, outputting the category information of the generated text.

[0005] During the implementation of the embodiments of the present disclosure, it was found that at least the following problems exist in the related art:

[0006] Related technologies use real-time and historical audio and video data as reference data to generate text for interactive scenarios. While real-time and historical audio and video data contain text features, multimodal data sources are not limited to audio and video data. Therefore, using only audio and video data as reference data for text extraction fails to fully exploit the short text features from other data sources within multimodal data, thus limiting the effectiveness of short text extraction.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field.

[0008] Summary of the Invention

[0009] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0010] The embodiments of the present disclosure provide a text classification method, apparatus, refrigeration equipment, and medium for multimodal data, so as to improve the effectiveness of short text extraction from multimodal data.

[0011] In some embodiments, the method includes: obtaining text information, voice information and image information based on multimodal data; obtaining voice text feature vectors and image text feature vectors based on the voice information and image information; performing text preprocessing on the text information, voice text feature vectors and image text feature vectors to obtain short text data.

[0012] In some embodiments, obtaining a speech-text feature vector and an image-text feature vector based on speech information and image information includes: preprocessing the speech information to obtain valid speech data and valid video data; performing speech recognition processing on the valid speech data to obtain a speech-text feature vector; and obtaining an image-text feature vector based on the valid video data and image information.

[0013] In some embodiments, obtaining the image text feature vector based on the valid video data and image information includes: performing frame extraction processing on the valid video data to obtain second image data; performing filtering processing on the image information to obtain first image data; performing image synthesis processing on the first image data and the second image data to obtain image data; and inputting the image data into a first deep neural network model for model training to obtain the image text feature vector.

[0014] In some embodiments, the first deep neural network model includes a migration and distillation diffusion deep fusion model.

[0015] In some embodiments, the speech recognition processing of the valid speech data to obtain the speech text feature vector includes: inputting the valid speech data to the encoder for speech encoding to obtain speech encoding data; inputting the speech encoding data to the decoder for speech decoding to obtain the speech text feature vector.

[0016] In some embodiments, the text information, speech text feature vectors and image text feature vectors are subjected to text preprocessing to obtain short text data, including: extracting the short text information of the text information, speech text feature vectors and image text feature vectors; and aggregating the short text information of the text information, speech text feature vectors and image text feature vectors to obtain short text data.

[0017] In some embodiments, the method further includes: performing data labeling processing on the short text data to obtain the data to be inspected; when the data to be inspected meets the quality inspection requirements, using the trained Bert model to extract text features on the data to be inspected to obtain target short text feature data; using the Focal Loss loss function to perform short text classification processing on the target short text feature data to obtain the target short text data.

[0018] In some embodiments, the BERT model is trained in the following manner: the initial BERT model is trained with the benchmark network parameters to obtain the network parameters and sorting information after the model training; the target number of network parameters is determined based on the network parameters and sorting information after the model training; the initial BERT model is trained with the target number of network parameters to obtain a trained BERT model.

[0019] In some embodiments, the method of training the initial Bert model using the target number of network parameters to obtain a trained Bert model includes: training the initial Bert model using the target number of network parameters to obtain a baseline Bert model; and migrating the teacher network model to the student network model using knowledge distillation technology to obtain a baseline student network model.

[0020] In some embodiments, the method of migrating the teacher network model to the student network model using knowledge distillation technology to obtain a baseline student network model further includes: obtaining the number of layers of all encoders of the baseline student network model and the weight of each encoder; determining the target number of layers of the encoder based on the number of layers of all encoders and the weight of each encoder; and training the baseline student network model using the encoder with the target number of layers to obtain a trained Bert model.

[0021] In some embodiments, the device includes: a speech preprocessing unit, configured to preprocess speech information associated with multimodal data to obtain valid speech data; a model training unit, configured to use a first deep neural network model to perform model training on image data to obtain an image text feature vector; a speech recognition unit, configured to perform speech recognition processing on valid speech data to obtain a speech text feature vector; a text preprocessing unit, configured to perform text preprocessing on text information, speech text feature vectors and image text feature vectors to obtain short text data.

[0022] In some embodiments, the voice preprocessing unit is also configured to preprocess the audio and video information to obtain valid video data, and the device also includes: an image frame extraction unit, configured to perform frame extraction processing on the valid video data to obtain second image data; an image filtering unit, configured to filter processing on the image information to obtain first image data; and an image fusion unit, configured to perform image synthesis processing on the first image data and the second image data to obtain image data.

[0023] In some embodiments, the device also includes: a data labeling unit, configured to perform data labeling processing on short text data to obtain data to be inspected; a data quality inspection unit, configured to use the trained Bert model to extract text features on the data to be inspected when the data to be inspected meets the quality inspection requirements to obtain target short text feature data; a text classification unit, configured to use the Focal Loss loss function to perform short text classification processing on the target short text feature data to obtain target short text data.

[0024] In some embodiments, the apparatus includes: a processor and a memory storing program instructions, wherein the processor is configured to execute the aforementioned text classification method for multimodal data when running the program instructions.

[0025] In some embodiments, the refrigeration equipment includes: an equipment body; and the aforementioned text classification device for multimodal data is installed in the equipment body.

[0026] In some embodiments, the storage medium stores program instructions, and when the program instructions are run, the program instructions execute the aforementioned text classification method for multimodal data.

[0027] The text classification method, apparatus, refrigeration equipment, and storage medium for multimodal data provided by the embodiments of the present disclosure can achieve the following technical effects:

[0028] The embodiment of the present disclosure obtains text information, voice information and image information based on multimodal data. Then, based on the voice information and image information, a voice text feature vector and an image text feature vector are obtained, so as to respectively reflect the text features of the voice and image using the voice text feature vector and the image text feature vector. Finally, the text information, the voice text feature vector and the image text feature vector are preprocessed to obtain short text data. In this way, by preprocessing the voice text feature vector and the image text feature vector that carry text features at the same time with the text information, the embodiment of the present disclosure can extract short text features from different data sources from the multimodal data, thereby obtaining high-quality short texts, thereby improving the effectiveness of short text extraction from multimodal data.

[0029] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] One or more embodiments are exemplarily described by corresponding drawings. These exemplary descriptions and drawings do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation. In addition,

[0031] FIG1 is a schematic diagram of a text classification apparatus for multimodal data provided by an embodiment of the present disclosure;

[0032] FIG2 is a schematic diagram of a text classification method for multimodal data provided by an embodiment of the present disclosure;

[0033] FIG3 is a schematic diagram of another text classification method for multimodal data provided by an embodiment of the present disclosure;

[0034] FIG4 is a schematic diagram of another text classification method for multimodal data provided by an embodiment of the present disclosure;

[0035] FIG5 is a schematic diagram of another text classification method for multimodal data provided by an embodiment of the present disclosure;

[0036] FIG6 is a schematic diagram of another text classification method for multimodal data provided by an embodiment of the present disclosure;

[0037] FIG7 is a schematic diagram of an application of an embodiment of the present disclosure;

[0038] FIG8 is a schematic diagram of a text classification device for multimodal data provided by an embodiment of the present disclosure;

[0039] FIG9 is a schematic diagram of another text classification apparatus for multimodal data provided by an embodiment of the present disclosure;

[0040] FIG10 is a schematic diagram of a refrigeration device provided in an embodiment of the present disclosure.

[0041] Reference numerals:

[0042] 10: Speech preprocessing unit; 20: Image filtering unit; 30: Speech recognition unit; 40: Image frame extraction unit;

[0043] 50: Image fusion unit; 60: Model training unit; 70: Text preprocessing unit; 80: Data annotation unit;

[0044] 90: Data quality inspection unit; 100: Text classification unit;

[0045] 200: a text classification device for multimodal data; 300: a text classification device for multimodal data;

[0046] 400: processor; 401: memory; 402: communication interface; 403: bus; 600: refrigeration equipment. DETAILED DESCRIPTION

[0047] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The accompanying drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0048] In the description and claims of the embodiments of the present disclosure, as well as in the accompanying drawings, the terms "first," "second," and the like are used to distinguish similar items and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate to describe the embodiments of the present disclosure herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions.

[0049] Unless otherwise stated, the term "plurality" means two or more.

[0050] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0051] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0052] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0053] As shown in Figures 1 and 8, an embodiment of the present disclosure provides a text classification device 200 for multimodal data, including a speech preprocessing unit 10, a model training unit 60, a speech recognition unit 30, and a text preprocessing unit 70. The speech preprocessing unit 10 is configured to preprocess the speech information associated with the multimodal data to obtain valid speech data. The model training unit 60 is configured to perform model training on the image data using a first deep neural network model to obtain an image-text feature vector. The speech recognition unit 30 is configured to perform speech recognition processing on the valid speech data to obtain a speech-text feature vector. The text preprocessing unit 70 is configured to perform text preprocessing on the text information, the speech-text feature vector, and the image-text feature vector to obtain short text data.

[0054] Optionally, the speech preprocessing unit 10 is further configured to preprocess the audio and video information to obtain valid video data. The text classification system for multimodal data also includes an image frame extraction unit 40, an image filtering unit 20, and an image fusion unit 50. The image frame extraction unit 40 is configured to perform frame extraction processing on the valid video data to obtain second image data. The image filtering unit 20 is configured to filter the image information to obtain first image data. The image fusion unit 50 is configured to perform image synthesis processing on the first image data and the second image data to obtain image data.

[0055] Optionally, the text classification system for multimodal data further includes a data annotation unit 80, a data quality inspection unit 90, and a text classification unit 100. The data annotation unit 80 is configured to perform data annotation processing on the short text data to obtain data to be inspected. The data quality inspection unit 90 is configured to, if the data to be inspected meets the quality inspection requirements, use a trained Bert model to extract text features from the data to be inspected to obtain target short text feature data. The text classification unit 100 is configured to perform short text classification processing on the target short text feature data using a Focal Loss loss function to obtain target short text data.

[0056] Based on the above-mentioned text classification system for multimodal data, in combination with FIG2 , an embodiment of the present disclosure provides a text classification method for multimodal data, including:

[0057] S01, the processor obtains text information, voice information and image information based on multimodal data.

[0058] In this step, the types of multimodal data include part or all of text information, voice information, and image information. The processor obtains multimodal data in the following manner: the processor obtains multimodal data through a human-computer interaction mode. The human-computer interaction mode includes but is not limited to Bluetooth, Wi-Fi (Wireless Fidelity), Internet, Web (World Wide Web), microphone array, microphone, and software. Among them, the software includes one or more of APP (Application), information collection tools, and touch screen.

[0059] S02: The processor obtains a speech text feature vector and an image text feature vector according to the speech information and the image information.

[0060] S03, the processor performs text preprocessing on the text information, the speech text feature vector and the image text feature vector to obtain short text data.

[0061] Using the text classification method for multimodal data provided by the embodiment of the present disclosure, the embodiment of the present disclosure obtains text information, voice information and image information based on the multimodal data. Then, based on the voice information and image information, a voice text feature vector and an image text feature vector are obtained, so as to respectively reflect the text features of the voice and image using the voice text feature vector and the image text feature vector. Finally, the text information, the voice text feature vector and the image text feature vector are preprocessed to obtain short text data. In this way, by preprocessing the voice text feature vector and the image text feature vector that carry text features at the same time with the text information, short text features from different data sources can be extracted from the multimodal data, thereby obtaining high-quality short texts, thereby improving the effectiveness of short text extraction from multimodal data.

[0062] Optionally, the processor extracts text information from the multimodal data in the following manner: the processor obtains initial text data from the multimodal data. The processor performs text preprocessing on the initial text data to obtain text information. The text preprocessing includes word segmentation, word form normalization, and stop word removal.

[0063] Optionally, as shown in FIG3 , the processor obtains a speech text feature vector and an image text feature vector according to the speech information and the image information, including:

[0064] S11, the processor pre-processes the voice information to obtain valid voice data and valid video data.

[0065] In this step, the processor preprocesses the voice information to obtain valid voice data and valid video data, including:

[0066] The processor performs data cleaning and format conversion on the multimodal data in sequence to extract valid sound data. The processor performs voice and video separation on the valid sound data to obtain voice preprocessing data and valid video data. The processor performs effective duration processing on the voice preprocessing data to obtain valid voice data. In this way, by performing data cleaning and format conversion processing, some interference in the multimodal data can be removed and the data format can be unified to obtain valid sound data. By performing voice and video separation processing on the valid sound data, the voice preprocessing data and the valid video data can be separated to obtain voice preprocessing data. By performing effective duration processing on the voice preprocessing data, empty voice and background sound can be effectively processed or eliminated. Among them, background sound includes machine sound during the machine operation stage. In actual applications, the processor can perform data cleaning, format conversion processing and voice and video separation processing on the multimodal data through data processing tools or data processing scripts.

[0067] S12, the processor performs speech recognition processing on the valid speech data to obtain a speech text feature vector.

[0068] In this step, the processor performs speech recognition processing on the valid speech data to obtain a speech text feature vector, including: the processor inputs the valid speech data to the first encoder for encoding processing to obtain speech coding data. The processor uses the second deep neural network model to extract features from the speech coding data to obtain a speech text feature vector. Optionally, the first encoder is a Wav2VEC encoder. The second deep neural network is configured with a multi-head attention mechanism. In this way, by adopting the second deep neural network configured with a multi-head attention mechanism, the mapping of the vector space of the speech coding data is realized, and the semantic, grammatical and contextual features covered by the speech are extracted and reflected through the speech text feature vector.

[0069] S13: The processor obtains an image text feature vector based on the valid video data and image information.

[0070] In this way, the embodiment of the present disclosure can extract valid voice data and valid video data from the voice information by preprocessing the voice information, reduce the interference in the multimodal data and achieve the unification of the data format. In addition, empty voice and background sound can be effectively eliminated. The effectiveness of the recognition of valid voice data and valid video data is improved. On this basis, the embodiment of the present disclosure further performs voice recognition processing on the valid voice data to obtain a voice-text feature vector. This is conducive to fully mining the voice-text features from the voice. And based on the valid video data and image information, an image-text feature vector is obtained, which is conducive to fully mining the image-text features from the image.

[0071] Optionally, as shown in FIG4 , the processor obtains an image text feature vector based on valid video data and image information, including:

[0072] S21: The processor performs frame extraction processing on the valid video data to obtain second image data.

[0073] S22: The processor filters the image information to obtain first image data.

[0074] S23: The processor performs image synthesis processing on the first image data and the second image data to obtain image data.

[0075] S24, the processor inputs the image data into the first deep neural network model for model training to obtain the image text feature vector.

[0076] Thus, the disclosed embodiment first extracts the second image data from the valid video data by performing frame extraction, and then filters the image information to extract the first image data from the image information. The first and second image data are then subjected to image synthesis processing to effectively fuse the first and second images, generating image data. Finally, the fused image data is input into a first deep neural network model for model training, which facilitates accurate recognition of semantic features from the image data.

[0077] Optionally, the first deep neural network model includes a migration and distillation diffusion deep fusion model.

[0078] The transfer and distillation diffusion deep fusion model uses transfer learning to train a simple network model using the output of a pre-trained complex model as a supervisory signal. Typically, the complex model is called the teacher model, and the simple network model is called the student model.

[0079] In this way, by selecting the migration and distillation diffusion deep fusion model as the first deep neural network model, the temporal, spatial and long-distance dependency characteristics of the model can be utilized to introduce the teacher model into the temporally and spatially continuous student model, which is more conducive to mining rich semantic features from image data and further improving the effectiveness of image text feature vectors.

[0080] Optionally, the processor performs speech recognition processing on the valid speech data to obtain a speech text feature vector, including:

[0081] Input valid voice data to the encoder for voice encoding to obtain voice encoded data.

[0082] Input the speech coding data to the decoder for speech decoding to obtain the speech text feature vector.

[0083] In this way, the embodiment of the present disclosure performs speech encoding by inputting valid speech data into the encoder, so that the speech encoding data obtained by encoding does not lose useful speech data, which is beneficial to the subsequent extraction of speech text features.

[0084] Optionally, the processor inputs the valid voice data to the encoder for voice encoding to obtain the coded voice data, including: the processor encodes the valid voice data using a Wav2VEC encoder to obtain the coded voice data. Thus, by encoding the valid voice data using the Wav2VEC encoder, the effect of eliminating redundant voice data can be enhanced, thereby obtaining purer coded voice data.

[0085] Optionally, the processor inputs the speech encoding data to the decoder for speech decoding to obtain a speech text feature vector, which includes: the processor inputs the speech encoding signal to a Transformer network model for decoding processing to obtain the speech text feature vector. In this way, using the Transformer network model to decode the speech encoding signal can further enhance the effect of eliminating redundant speech data, thereby fully extracting speech text features.

[0086] Optionally, the processor performs text preprocessing on the text information, the speech text feature vector, and the image text feature vector to obtain short text data, including:

[0087] The processor extracts short text information of each of the text information, the speech text feature vector, and the image text feature vector.

[0088] The processor collects the short text information of the text information, the speech text feature vector, and the image text feature vector to obtain short text data.

[0089] Thus, the disclosed embodiment extracts short text information from text information, speech text feature vectors, and image text feature vectors, respectively, and then aggregates the short text information from these different data sources. Effectively extracting short text features from different data sources facilitates obtaining high-quality short text, thereby improving the effectiveness of short text extraction from multimodal data.

[0090] As shown in FIG5 , the present disclosure also provides a method for text classification of multimodal data, including:

[0091] S31: The processor obtains text information, voice information, and image information based on the multimodal data.

[0092] S32: The processor obtains a speech text feature vector and an image text feature vector according to the speech information and the image information.

[0093] S33, the processor performs text preprocessing on the text information, the speech text feature vector, and the image text feature vector to obtain short text data.

[0094] S34, the processor performs data annotation processing on the short text data to obtain data to be quality inspected.

[0095] In this step, the processor performs data labeling on the short text data to obtain the data to be inspected, including: the processor labels the short text data with slots and metadata attribute tags to obtain the data to be inspected. This provides a basis for subsequent data quality inspection.

[0096] S35: When the data to be inspected meets the quality inspection requirements, the processor uses the trained Bert model to extract text features from the data to be inspected to obtain target short text feature data.

[0097] In this step, the processor determines that the data to be processed meets the quality inspection requirements in the following manner: the metadata attribute labels and the food attribute labels are correct and the error rate is less than or equal to 2%.

[0098] S36: The processor performs short text classification processing on the target short text features using the Focal Loss function to obtain target short text data.

[0099] The text classification method for multimodal data provided by the embodiment of the present disclosure adopts the method. After obtaining the short text data, the embodiment of the present disclosure sequentially performs data labeling and data quality inspection. When the quality inspection requirements are met, the trained Bert model is used to extract text features from the data to be inspected to obtain target short text feature data, thereby facilitating the extraction of short text feature data containing rich short text features from the data to be inspected. Due to the problem of type imbalance in short text feature data, the classification accuracy of the short text classification process is affected. To this end, the embodiment of the present disclosure uses the Focal Loss loss function to perform short text classification processing on the target short text features to reduce the impact of the type imbalance of the short text feature data on the accuracy of the short text classification process, thereby further ensuring the quality of the short text.

[0100] Optionally, the processor uses a Focal Loss loss function to perform short text classification processing on the target short text features to obtain target short text data, including: the processor uses a Focal Loss loss function to perform short text classification processing on the target short text feature data to obtain short text classification data. The processor performs data enhancement processing on the short text classification data to obtain target short text data. In this way, by performing data enhancement processing on the short text classification data, a limited short text classification data set can be expanded, the training effect of the BERT model can be improved, and the generalization ability of the model can be enhanced.

[0101] Optionally, in conjunction with FIG6 , the processor trains the BERT model in the following manner:

[0102] S41, the processor performs model training on the initial Bert model using the benchmark network parameters to obtain the network parameters and ranking information after model training.

[0103] S42: The processor determines a target number of network parameters based on the network parameters after model training and the sorting information.

[0104] In this step, the processor determines the target number of network parameters based on the trained network parameters and the ranking information, including: the processor selects the number of network parameters α before ranking as the target number based on the ranking information. The processor calculates N = α × N0, where 0.5 ≤ α ≤ 0.8. Where N0 represents the total number of network parameters. N represents the target number of network parameters. α represents the first weight value. Preferably, α = 0.75.

[0105] S43: The processor trains the initial BERT model using the target number of network parameters to obtain a trained BERT model.

[0106] In this way, the number of network parameters of the initial BERT model is intercepted to achieve fine-tuning of the initial BERT model, retaining the number of network layers and network parameters with high model weights, which can reduce the memory space occupied by the model.

[0107] Optionally, the processor obtains the baseline network parameters in the following manner:

[0108] The processor extracts training data from the target short text data.

[0109] The processor uses the training data to perform model training on the initial BERT model to obtain benchmark network parameters.

[0110] This is helpful to improve the accuracy of the baseline network parameters and improve the training effect of the Bert model.

[0111] Optionally, the processor trains an initial BERT model using a target number of network parameters to obtain a trained BERT model, including:

[0112] The processor trains an initial BERT model using a target number of network parameters to obtain a baseline BERT model. The processor then uses knowledge distillation technology to migrate the teacher network model to the student network model, obtaining a baseline student network model. The baseline BERT model is the teacher network model, and the baseline student network model is the trained BERT model. By migrating the teacher network model to the student network model, the large model guides the training of the small model. The resulting baseline student network model achieves performance comparable to that of the teacher network model, while significantly reducing the number of network parameters and achieving model compression and acceleration.

[0113] Optionally, after the processor uses knowledge distillation technology to transfer the teacher network model to the student network model to obtain a baseline student network model, the process further includes: obtaining the number of layers of all encoders of the baseline student network model and the weight of each encoder. The processor determines a target number of encoder layers based on the number of layers of all encoders and the weight of each encoder. The processor trains the baseline student network model using the encoders with the target number of layers to obtain a trained BERT model.

[0114] In this way, the number of encoder layers of the pruned Bert model is reduced, which can reduce the model's memory space occupation and improve the efficiency of the Bert model training.

[0115] Optionally, the processor determines a target number of encoder layers based on the number of layers of all encoders and the weight of each encoder, including: the processor sorts the weights of all encoders in descending order according to their weights. The processor selects the number of layers of the encoders with the first β positions in the descending order as the target number of layers. The processor calculates M = β × M0. 0.5≤β≤0.8. Where M0 represents the number of layers of all encoders. M represents the target number of layers of the encoder. β represents the second weight value. Preferably, β = 0.75.

[0116] In practical applications, as shown in Figure 7, the text classification method for multimodal data includes the following steps:

[0117] S101: The processor collects multimodal data.

[0118] S102: The processor cleans the multimodal data to obtain voice information, text information, and image information.

[0119] S103: The processor pre-processes the voice information to obtain valid voice data and valid video data.

[0120] S1041: The processor inputs valid voice data and passes it through the Wav2 encoder and Transformer decoder in sequence to obtain a voice text feature vector.

[0121] S1042: The processor extracts image frames from the valid video data to obtain a second target image.

[0122] S1043: The processor performs image filtering on the image information to obtain a first target image.

[0123] S1044: The processor performs image synthesis on the second target image and the first target image to obtain image data.

[0124] S1045: The processor inputs the image data into the migration and distillation diffusion deep fusion model to perform vector space mapping processing to extract the image text feature vector from the image data.

[0125] S105: The processor performs text preprocessing on the text information, the speech text feature vector, and the image text feature vector to obtain short text data.

[0126] S106: The processor performs data annotation processing on the short text data to obtain data to be quality inspected.

[0127] S107: The processor determines whether the data to be inspected meets the quality inspection requirements. If so, execute S108; otherwise, execute S109.

[0128] S108: The processor outputs the data to be inspected as a text classification result.

[0129] S109: The processor uses the trained Bert model to extract text features from the quality inspection data to obtain target short text feature data.

[0130] S110: The processor performs short text classification processing on the target short text features using the Focal Loss function to obtain target short text data, and outputs the target short text data as a text classification result.

[0131] As shown in Figure 9, an embodiment of the present disclosure provides a text classification device 300 for multimodal data, including a processor 400 and a memory 401. Optionally, the device may also include a communication interface 402 and a bus 403. The processor 400, the communication interface 402, and the memory 401 can communicate with each other through the bus 403. The communication interface 402 can be used for information transmission. The processor 400 can call the logic instructions in the memory 401 to execute the text classification method for multimodal data of the above embodiment.

[0132] In addition, the logic instructions in the memory 401 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0133] Memory 401, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present disclosure. Processor 400 executes the program instructions / modules stored in memory 401 to perform functional applications and data processing, thereby implementing the text classification method for multimodal data in the above-mentioned embodiments.

[0134] The memory 401 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the use of the terminal device. Furthermore, the memory 401 may include high-speed random access memory and non-volatile memory.

[0135] As shown in FIG10 , an embodiment of the present disclosure provides a refrigeration device 600, comprising: a device body, and the above-mentioned text classification device 200 (300) for multimodal data. The text classification device 200 (300) for multimodal data is installed on the device body. The installation relationship described here is not limited to placement inside the product, but also includes installation connections with other components of the product, including but not limited to physical connections, electrical connections, or signal transmission connections. It can be understood by those skilled in the art that the text classification device 200 (300) for multimodal data can be adapted to a feasible product body, thereby realizing other feasible embodiments.

[0136] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned text classification method for multimodal data.

[0137] An embodiment of the present disclosure provides a computer program, which, when executed by a computer, enables the computer to implement the above-mentioned text classification method for multimodal data.

[0138] An embodiment of the present disclosure provides a computer program product, which includes computer instructions stored on a computer-readable storage medium. When the program instructions are executed by a computer, the computer implements the above-mentioned text classification method for multimodal data.

[0139] The aforementioned computer-readable storage medium may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0140] The technical solution of the embodiments of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code, or a transient storage medium.

[0141] The reference to any prior art in the specification is not and should not be taken as an admission or any form of suggestion that the prior art forms part of the common general knowledge in the application area or any other jurisdiction, or that the prior art could reasonably be understood and regarded as relevant by a person skilled in the art.

[0142] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to also include plural forms. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variations "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, an element defined by the sentence "comprising a..." does not exclude the presence of other identical elements in the process, method or device that includes the element. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments can be referenced to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can be found in the description of the method part.

[0143] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units can be merely a logical functional division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, and can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of the present disclosure may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0145] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the systems, methods and computer program products according to the embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of the code, and the module, program segment or part of the code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A text classification method for multimodal data, characterized in that: include: According to the multimodal data, text information, voice information and image information are obtained; According to the voice information and the image information, a voice text feature vector and an image text feature vector are obtained; The text information, speech text feature vector and image text feature vector are preprocessed to obtain short text data.

2. The method according to claim 1, wherein: The step of obtaining a speech text feature vector and an image text feature vector according to the speech information and the image information includes: Preprocess the voice information to obtain valid voice data and valid video data; Perform speech recognition processing on valid speech data to obtain speech text feature vectors; According to the effective video data and image information, the image text feature vector is obtained.

3. The method according to claim 2, wherein: The step of obtaining the image text feature vector according to the valid video data and the image information includes: Performing frame extraction processing on the effective video data to obtain second image data; Filtering the image information to obtain first image data; Performing image synthesis processing on the first image data and the second image data to obtain image data; Input the image data into the first deep neural network model for model training to obtain the image text feature vector.

4. The method according to claim 3, wherein: The first deep neural network model includes a migration and distillation diffusion deep fusion model.

5. The method according to claim 2, wherein: The performing speech recognition processing on the effective speech data to obtain the speech text feature vector includes: Input valid voice data to the encoder for voice encoding to obtain voice encoding data; Input the speech coding data to the decoder for speech decoding to obtain the speech text feature vector.

6. The method according to any one of claims 1 to 5, wherein: The text information, the speech text feature vector and the image text feature vector are subjected to text preprocessing to obtain short text data, including: Extracting short text information of the text information, the speech text feature vector and the image text feature vector; The short text information of the text information, the speech text feature vector and the image text feature vector are collected to obtain short text data.

7. The method according to any one of claims 1 to 6, wherein: Also includes: Perform data annotation processing on short text data to obtain data to be inspected; When the quality inspection data meets the quality inspection requirements, the trained Bert model is used to treat the quality inspection data Perform text feature extraction to obtain target short text feature data; The Focal Loss loss function is used to perform short text classification on the target short text feature data to obtain the target short text data.

8. The method according to claim 7, wherein: Train the BERT model as follows: The initial Bert model is trained with the benchmark network parameters to obtain the network parameters and ranking information after model training; Determine the target number of network parameters based on the network parameters and ranking information after model training; The initial BERT model is trained using the target number of network parameters to obtain a trained BERT model.

9. The method according to claim 8, wherein: The method of training the initial Bert model using the target number of network parameters to obtain the trained Bert model includes: The initial Bert model is trained using the target number of network parameters to obtain a benchmark Bert model; The knowledge distillation technology is used to transfer the teacher network model to the student network model to obtain the benchmark student network model.

10. The method according to claim 9, wherein: After the teacher network model is transferred to the student network model by using the knowledge distillation technology to obtain the benchmark student network model, the method further includes: Get the number of layers of all encoders and the weights of each encoder of the benchmark student network model; Determine the target number of encoder layers based on the number of layers of all encoders and the weight of each encoder; The benchmark student network model is trained using an encoder with the target number of layers to obtain a trained BERT model.

11. A text classification device for multimodal data, characterized in that: include: A speech preprocessing unit, configured to preprocess the speech information associated with the multimodal data to obtain valid speech data; A model training unit is configured to perform model training on the image data using the first deep neural network model to obtain an image text feature vector; A speech recognition unit is configured to perform speech recognition processing on valid speech data to obtain a speech text feature vector; The text preprocessing unit is configured to perform text preprocessing on text information, speech text feature vectors and image text feature vectors to obtain short text data.

12. The device according to claim 11, wherein The speech preprocessing unit is further configured to preprocess the audio and video information to obtain valid video data. The device also includes: An image frame extraction unit is configured to perform frame extraction processing on the effective video data to obtain second image data; An image filtering unit, configured to filter the image information to obtain first image data; The image fusion unit is configured to perform image synthesis processing on the first image data and the second image data. Obtain image data.

13. The device according to claim 11, wherein: The device also includes: A data annotation unit is configured to perform data annotation processing on the short text data to obtain data to be inspected; The data quality inspection unit is configured to extract text features from the data to be inspected using the trained Bert model to obtain target short text feature data when the data to be inspected meets the quality inspection requirements; The text classification unit is configured to perform short text classification processing on the target short text feature data using a Focal Loss loss function to obtain target short text data.

14. A text classification device for multimodal data, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the text classification method for multimodal data according to any one of claims 1 to 10 when running the program instructions.

15. A refrigeration device, characterized in that: include: Equipment body; The apparatus for multimodal data text classification as described in any one of claims 11 to 14 is installed on the device body.

16. A storage medium storing program instructions, characterized in that: When the program instructions are executed, the text classification method for multimodal data according to any one of claims 1 to 10 is executed.

17. A computer program, when executed by a computer, enables the computer to implement the text classification method for multimodal data according to any one of claims 1 to 10.

18. A computer program product, comprising computer instructions stored on a computer-readable storage medium, wherein when the program instructions are executed by a computer, the computer is enabled to implement the text classification method for multimodal data as described in any one of claims 1 to 10.

Images