Text classification method and refrigeration equipment system

Abstract

The application discloses a text classification method and a refrigeration equipment system, and the method comprises the following steps: all multi-modal data and / or multi-source heterogeneous data are transcribed into text data and are summarized as total text data; the text in the total text data that can be labeled is labeled; the total text data is subjected to feature extraction through a deep fusion network model; and a matching result of text classification is calculated based on the extracted features. The text classification method fully utilizes multi-modal and multi-source heterogeneous data, fully mines semantic, grammatical and contextual information of natural language understanding of the data itself and between the data, has high text classification accuracy, improves the value of the data, improves the accuracy of user feedback information, and provides a better prospect for application and popularization of artificial intelligence on refrigeration equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of refrigeration equipment, and in particular to a text classification method and a refrigeration equipment system. BACKGROUND

[0002] With the progress of artificial intelligence, people hope to introduce artificial intelligence into the field of refrigerators to make refrigerators more intelligent. In the process of intelligentization of refrigerators, a large number of optimization processes for refrigerator scenarios are involved, and such optimization includes optimization of various types of interactions between users and the refrigerator in spoken language, text, and video. If the optimization effect is not good and the refrigerator cannot accurately answer the user's questions, the refrigerator will not be able to meet the user's use expectations, the user's use frequency of intelligent scenarios of the refrigerator will be low, and the user's interaction experience will not be good. In the process of implementing the present application, the inventors found that the main reason for the poor optimization effect is the low accuracy of text classification, and the reason for the low accuracy of text classification is that the prior art only considers single text data without combining the multi-modal data interaction process between the user and the refrigerator. SUMMARY

[0003] To solve at least one of the above-mentioned prior art problems, the purpose of the present application is to provide a text classification method and a refrigeration equipment system.

[0004] To achieve the above-mentioned application purpose, an embodiment of the present application provides a text classification method, comprising the following steps:

[0005] Transcribe all multi-modal data and / or multi-source heterogeneous data into text data and aggregate them into total text data;

[0006] Label the labelable text in the total text data, and store the labeled text as training data and testing data, respectively;

[0007] Extract features from the total text data through a deep fusion network model, train the deep fusion network model using the training data, and predict the result through the deep fusion network model using the testing data, wherein the deep fusion network model is a fusion model of a text vectorization model and a multi-dimensional feature extraction model, the text vectorization model vectorizes the text data, and the multi-dimensional feature extraction model extracts multi-dimensional interaction features and correlation features;

[0008] Calculate the matching result of text classification based on the extracted features.

[0009] As a further improvement of the present application, the step of labeling the labelable text in the total text data comprises:

[0010] The markable text is sequentially pre-labeled, formally labeled and labeled quality inspection, wherein when the value of the text after formal labeling is lower than the preset threshold in the labeled quality inspection judgment, the text is returned to the pre-labeling for re-labeling.

[0011] To achieve one of the above-mentioned purposes, an embodiment of the present application provides a text classification method, comprising the following steps:

[0012] All multi-modal data and / or multi-source heterogeneous data are transcribed into text data and summarized as total text data;

[0013] The text in the total text data that cannot be labeled is data cleaned;

[0014] The total text data is extracted by a deep fusion network model, and an unsupervised learning algorithm is used to train the deep fusion network model with the unlabelable text, the deep fusion network model is a fusion model of a text vectorization model and a multi-dimensional feature extraction model, the text vectorization model vectorizes the text data, and the multi-dimensional feature extraction model extracts multi-dimensional interaction features and correlation features;

[0015] Based on the extracted features, the matching result of text classification is calculated.

[0016] As a further improvement of the present application, the step of transcribing all multi-modal data and / or multi-source heterogeneous data into text data and summarizing them as total text data comprises:

[0017] Collecting multi-modal data and / or multi-source heterogeneous data and preprocessing, wherein the multi-modal data includes text, audio and video data;

[0018] Transcribe the video data into text data;

[0019] Transcribe the audio data into text data;

[0020] Obtain historical text data;

[0021] The text data in the multi-modal data, the text data transcribed from the audio data, the text data transcribed from the video data, and the historical text data are summarized as the total text data.

[0022] As a further improvement of the present application, the preprocessing includes cleaning, format conversion and storage of the multi-modal data.

[0023] As a further improvement of the present application, the step of transcribing the video data into text data comprises:

[0024] Separate the audio and image in the video data to obtain audio data and image data;

[0025] recognize the character information in the image data and transcribe the character information into text data;

[0026] recognize the image data based on spatio-temporal and long-distance dependency features and transcribe the image data into text data.

[0027] As a further improvement of the present application, the step of recognizing the image based on spatio-temporal and long-distance dependency features and transcribing the image into text data comprises:

[0028] generating a student model based on the recognition of the image based on spatio-temporal and long-distance dependency features through a fusion model of a knowledge distillation and diffusion model, and transcribing the image data into text data through the student model.

[0029] As a further improvement of the present application, the step of transcribing the video data into text data comprises:

[0030] obtaining key frame images in the video data through a diffusion network model;

[0031] recognizing the key frame images to generate text data.

[0032] As a further improvement of the present application, the step of transcribing the audio data into text data comprises:

[0033] combining the speech spatio-temporal characteristics and context relationship features in the data acquisition scenario to establish a deep recurrent convolutional network model based on a fusion neural network MMCNN-RNN, CTC, and Attention to transcribe the audio data into text data.

[0034] As a further improvement of the present application, the historical text data comprises historical record data and historical interaction data, wherein the historical record data comprises user's food material preference data, interest data, and comment data, and the historical interaction data comprises interaction records obtained from a client or an interaction end of a refrigeration device.

[0035] As a further improvement of the present application, the text vectorization model encodes text at the word, phrase, sentence level.

[0036] As a further improvement of the present application, the multi-dimensional feature extraction model uses a multi-head attention mechanism to extract word, phrase, sentence interaction and correlation features, and context semantic information.

[0037] As a further improvement of the present application, the step of calculating the matching result of text classification based on the extracted features comprises:

[0038] the vectors after feature extraction are sequentially subjected to a fully connected layer and a self-attention mechanism to obtain a text classification result.

[0039] As a further improvement of the present application, further comprising the steps of:

[0040] The calculated text classification result is reached.

[0041] To achieve one of the above-mentioned purposes, an embodiment of the present application provides a refrigeration equipment system, comprising:

[0042] A storage module for storing a computer program;

[0043] A processing module for implementing the steps of the above-mentioned text classification method when executing the computer program.

[0044] Compared with the prior art, the present application has the following beneficial effects: the text classification method makes full use of multi-modal and multi-source heterogeneous data, and fully excavates the semantic, syntactic and contextual information of natural language understanding of the data itself and between the data, the accuracy of text classification is high, the value of data is improved, and the accuracy of user feedback information is improved, which provides a better prospect for the application and popularization of artificial intelligence on refrigeration equipment. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a structural schematic diagram of the refrigeration equipment system of an embodiment of the present application;

[0046] Figure 2 is a flowchart of the text classification method of an embodiment of the present application;

[0047] Figure 3 is a data flow diagram of the text classification method of an embodiment of the present application;

[0048] Figure 4 is a flowchart of the text classification method of the first embodiment of the present application;

[0049] Figure 5 is a flowchart of the text classification method of the second embodiment of the present application;

[0050] Figure 6 is a module schematic diagram of the text classification device of an embodiment of the present application;

[0051] Figure 7 is a structural block diagram of the refrigeration equipment system of an embodiment of the present application;

[0052] Among them, 100, refrigeration equipment; 10, interactive screen; 20, camera; 30, microphone; 40, loudspeaker; 50, processing module; 60, storage module; 70, communication bus; 200, client. DETAILED DESCRIPTION

[0053] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0054] One embodiment of the present invention provides a text classification method and a cooling device system. The method utilizes data from multiple modalities for thorough training, resulting in higher accuracy in text classification.

[0055] The refrigeration system may include a refrigeration device 100 and a client 200 corresponding to the refrigeration device 100. The refrigeration device 100 may be a refrigerator, and the client 200 may be a mobile phone or an app on a mobile phone. (Refrigeration system parameters...) Figure 1 As shown, the refrigeration device 100 and the client 200 can be connected via wireless signal. The following description uses a refrigerator as an example to illustrate the refrigeration device 100.

[0056] Continue as Figure 1 As shown, the refrigeration device 100 can be a refrigerator with audio acquisition, video acquisition, and user interface interaction. The refrigerator is equipped with a microphone 30 for audio acquisition, a camera 20 for video recording, a speaker 40 for voice interaction with the user, and an interactive screen 10 for text or graphical interface interaction with the user. The interactive screen 10 can be installed on the refrigerator door. After the user opens the refrigerator door, the camera 20 records the user's operation and forms video data. The speaker 40 and microphone 30 are combined to interact with the user in a question-and-answer format with audio.

[0057] Taking mobile phones as an example, the client 200 allows users to communicate with the refrigerator via text or voice, manage food information inside the refrigerator, and control the refrigerator's operating status.

[0058] In addition, the cooling system may include other external devices, such as external temperature sensors, cameras 20 or microphones 30 and speakers 40 built into other devices, smart speakers, etc. These devices can be connected to the cooling system 100 or the client 200 via wireless signals.

[0059] Data from various devices within the refrigeration system forms multi-source heterogeneous data. This multi-source heterogeneous data collected by multiple devices can be transmitted via wired, Wi-Fi, Bluetooth, and other methods. Various types of data, including text, audio, and video, constitute multimodal data. This data can be real-time online or offline, or it can be stored historical data.

[0060] The refrigeration equipment scene includes the interaction of the user with the refrigeration equipment 100, the interaction of the user with the corresponding client 200 of the refrigeration equipment 100, for example, the interaction of the food, the interaction of the instruction, the recording of the video when the user operates the refrigeration equipment 100, the control of the user on the internal temperature and humidity of the refrigeration equipment 100, and the review of the user on the food on the client 200, the preference of the user and the like, the data generated by the direct operation of the refrigeration equipment 100 and the data generated by the client 200 related to the refrigeration equipment 100 are all the data in the refrigeration equipment scene.

[0061] The embodiment fully mines the semantic, syntactic and contextual information of natural language understanding of the data itself and the data by using the multi-modal real-time and offline data generated in the refrigeration equipment use scene, the accumulated massive text historical data, so that the text classification result in the refrigeration equipment scene is more accurate.

[0062] The following will be combined Figure 2 and Figure 5 to describe a text classification method provided by an embodiment of the application. Although the application provides the method operation steps shown in the following embodiments or flowcharts, the execution order of the steps in the logical relationship without necessary causality is not limited to the execution order provided in the embodiments of the application based on conventional or non-creative labor. For example, the acquisition order of the following steps S20, S30 and S40 can be adjusted or performed at the same time, and the time sequence is not distinguished.

[0063] Specifically, the text classification method includes the following steps:

[0064] Step S100: All the multi-modal data and / or multi-source heterogeneous data are transcribed into text data and summarized as total text data.

[0065] The composition of the multi-modal data and / or multi-source heterogeneous data is described above, and the specific collection and transcription steps can include the following steps S10-S40.

[0066] Step S10: Collecting multi-modal and / or multi-source heterogeneous data and preprocessing, wherein the multi-modal data includes text, audio and video data related to the refrigeration equipment scene.

[0067] The preprocessing includes cleaning, format conversion and storage of the multi-modal data. The format conversion includes the analysis of the data format.

[0068] Text is collected through the interaction screen 10 of the client 200 and or refrigerator end, the client 200 can include mobile phones, pads, PC ends and other devices, in addition to the text data generated on the application program, it also includes customer service text, web text, and text on the applet or public number, the preprocessing of the text data can adopt stop words, deduplication and the like.

[0069] Audio data can be collected by the microphone 30 of the mobile phone end and or the refrigerator end, or connected to other sound collecting units through WiFi and the like, to obtain the voice of the sound collecting unit on other devices, the microphone 30 can be a single microphone 30 or a microphone 30 array.

[0070] Video data can be collected by the camera 20 of the client 200 and or the refrigerator end, for example, through the app of the mobile phone, the camera 20, Bluetooth and the like, in addition, the voice and video can be separated by a script, and then the effective voice and video data are obtained.

[0071] Through the above-mentioned various ways, the multi-modal data collection task of various channels and terminals and the like is completed, and the data integrity and multi-modal cognitive characteristics are ensured.

[0072] Most of the prior art only uses text as a single data, which leads to the neglect of other data in the later data learning, and further, the recognition accuracy is low when the user interacts with the refrigerator in other forms. The text source of the present application includes various modal data generated by the user, which is further trained and classified, and the accuracy is higher.

[0073] Step S20: converting the video data into text data.

[0074] This step includes the following two embodiments, in one of which, the following steps are included:

[0075] Separating the audio and image in the video data to obtain audio data and image data;

[0076] Recognizing the text information in the image data and converting it into text data;

[0077] Recognizing the image data based on space-time and long-distance dependence features and converting it into text data.

[0078] The recognition of the text in the image data can directly transcribe it into text data, and the recognition of the non-text in the image can be the recognition of the content said by the user according to the changes of the multiple images of the user's mouth movement. And the application considers that if only the image features of the speaker are used, the recognized sentence may be complex, so the application combines the sentence length factor and the context relevance for recognition, the sentence length factor includes the features of different sentence lengths and different word compositions, and the recognition is performed based on the space-time and long-distance dependence features to mine rich semantic feature information of the sentence sequence.

[0079] In addition, in order to speed up the response speed of the model, a student model is generated by fusing a knowledge distillation model and a diffusion model based on the space-time and long-distance dependence feature recognition image, the knowledge of the original large model is migrated to the student network, and the image data is transcribed into text data by the student model. The student model is a model with discrete time steps and short steps, and the student model can be distilled to half of the steps of the teacher model.

[0080] In another embodiment, the key frame image in the video data can be obtained by a diffusion model;

[0081] The key frame image is recognized to generate text data.

[0082] Here, the meaning of the image is recognized, and the content of the image is described in the form of text, and the transcription of the image data into text data is completed.

[0083] Step S30: The audio data is transcribed into text data by combining the speech space-time characteristics and the context relationship features of the refrigeration equipment 100.

[0084] The audio data of this step can be directly collected audio data, and can also include audio data segmented from video data.

[0085] This embodiment combines the speech data space-time characteristics of the use scene of the refrigeration equipment 100 such as a refrigerator, and establishes a deep recurrent convolutional network model based on a fusion neural network MMCNN-RNN, CTC, and Attention through an end-to-end learning method to transcribe the audio data into text data, so as to obtain rich high-level speech feature information and improve the accuracy of the model speech transcription text.

[0086] Step S40: Obtain historical text data, wherein the historical text data includes historical record data and historical interaction data.

[0087] The multi-modal data collected by the above step S10 is real-time data, and the historical text data obtained by the step S40 is historical data. By obtaining the historical text data, on the one hand, the information of the historical text data itself can be utilized, and on the other hand, the historical text data can be complementary and associated with the real-time data, so that the semantic information of the text data can be fully obtained.

[0088] The historical text data can be a lot of unlabeled texts accumulated on the refrigeration equipment 100 or the client 200. After obtaining the historical text data, the collected data can be uniformly processed, such as cleaning and format conversion, so that the text formats of the historical text data and the real-time collected data are unified, and the comprehensiveness and particularity of the data characteristics are ensured.

[0089] Further, the historical record data includes user's food material preference data, interest data and comment data; and the historical interaction data includes interaction records obtained from the interaction end of the client 200 or the refrigeration equipment 100. The historical text data also includes data collected by the refrigeration equipment 100 and data on the client 200 corresponding to the refrigeration equipment 100.

[0090] Step S50: preprocessing the total text data.

[0091] Step S60: performing feature extraction on the total text data by a deep fusion model, wherein the deep fusion model is a fusion model of a text vectorization model and a multi-dimensional feature extraction model.

[0092] The text data in the multi-modal data, the text data converted from the audio data, the text data converted from the video data, and the historical text data are summarized as total text data.

[0093] The total text data reflects that the embodiment adopts multi-source heterogeneous data, real-time, offline voice, video, image, text and user historical comments, food material preference, food material interest and other text data.

[0094] The deep fusion network model of the step S60 is a fusion model of a text vectorization model and a multi-dimensional feature extraction model, the text vectorization model vectorizes the text data, and the multi-dimensional feature extraction model extracts multi-dimensional interaction features and correlation features.

[0095] The total text data can include two types of data, the first type of data can be pre-labeled, and the second type of data cannot be labeled. The two types of data are processed in different ways in steps S50 and S60. The processing of the first type of data is embodiment 1, and the processing of the second type of data is embodiment 2.

[0096] Embodiment 1

[0097] Referring to FIG. 1, the step S50 and the step S60 respectively include: Figure 4 The step S51: annotating the annotatable text in the total text data, and storing the annotated text as training data and testing data respectively.

[0098] The step S51: annotating the annotatable text in the total text data, and storing the annotated text as training data and testing data respectively.

[0099] The step S61: extracting features of the total text data by using a deep fusion network model, training the deep fusion network model by using the training data, and predicting results by using the testing data through the deep fusion network model.

[0100] The step S51 can further include the steps of pre-annotation, formal annotation, annotation quality inspection and data storage for the annotatable text in sequence, wherein when the value of the text after formal annotation is lower than a preset threshold in the annotation quality inspection judgment, the text is returned to the pre-annotation for re-annotation. The annotated data can include training data and testing data, realizing supervised or semi-supervised in the subsequent step S61.

[0101] The annotated data in the step S51 can include training data and testing data, and the step S61 can perform a pre-training task on the constructed deep fusion model based on the training data, and then perform result prediction based on the testing data, so as to obtain rich semantic feature information, thereby obtaining the best effective model, ensuring that the prediction result is optimal, and the accuracy of user feedback information is higher.

[0102] Embodiment 2

[0103] Referring to FIG. 1, the step S50 and the step S60 respectively include: Figure 5 The step S51: annotating the annotatable text in the total text data, and storing the annotated text as training data and testing data respectively.

[0104] The step S52: data cleaning for the unannotatable text in the total text data.

[0105] The step S62: extracting features of the total text data by using a deep fusion network model, training the deep fusion network model by using the unannotatable text through an unsupervised learning algorithm.

[0106] In the step S52, the text data that cannot be annotated is directly subjected to data cleaning, format conversion and other tasks, and then directly participates in the subsequent step S62 without annotation. The data that cannot be annotated is subjected to text analysis in the subsequent step in the form of unsupervised or unannotated data.

[0107] The text vectorization model in the step S60 encodes the text at the word, phrase, sentence level. The multi-dimensional feature extraction model uses a multi-head attention mechanism to extract word, phrase, sentence interaction and correlation features, and context semantic information.

[0108] Step S70: based on the extracted features, the matching result of text classification is calculated to realize the text classification matching task.

[0109] In step S70, the vector after feature extraction is sequentially subjected to a fully connected layer and a self-attention mechanism to obtain a text classification result. Here, the classification result is first calculated by the fully connected layer, and then the semantic relationship quantization result with stronger text interaction is calculated by the self-attention mechanism.

[0110] In addition, step S70 can also use text similarity with ambiguity elimination for calculation, which is realized based on the threshold size control of the distance between word items, so that some redundant information in the text can be removed.

[0111] In order to optimize the parameters and weights of the model, the result of step S70 is returned to step S50 again, wherein steps S51 and S52 can be executed for labeling, and the cycle is run multiple times until the text classification effect is optimal.

[0112] Step S80: result delivery.

[0113] The text classification result after the above steps is run is delivered, and the result delivery method can adopt various forms with or without external delivery, such as external call, short message delivery, email notification, large screen display, voice broadcast, text output, smart speaker, pop-up UI, app end, PAD, web end, and various result delivery methods to meet the needs of result display and digital display.

[0114] Compared with the prior art, the embodiment has the following beneficial effects:

[0115] The text classification method fully utilizes multi-modal and multi-source heterogeneous data, and fully excavates the semantic, syntactic and contextual information of natural language understanding of the data itself and between the data, has high accuracy of text classification, improves the value of data, and improves the accuracy of user feedback information, providing a better prospect for the application and popularization of artificial intelligence on the refrigeration equipment 100.

[0116] In one embodiment, the present application also proposes a refrigeration equipment system comprising a storage module 60 and a processing module 50, wherein the processing module 50 can realize the steps of the above-mentioned text classification method when executing the computer program, that is, realize the steps in any one of the above-mentioned text classification methods.

[0117] The refrigeration equipment system can also comprise the following modules as shown in the figure, and the specific functions of each module are as follows: Figure 6

[0118] ​The collection module is configured to collect multi-modal data and / or multi-source heterogeneous data and pre-process the multi-modal data, wherein the multi-modal data comprises text, audio and video data related to a refrigeration equipment scene;

[0119] The video transcription module is configured to transcribe the video data into text data.

[0120] The audio transcription module is configured to transcribe the audio data into text data in combination with the voice spatiotemporal characteristics and context relationship features of the refrigeration equipment 100.

[0121] The acquisition module is configured to acquire historical text data, wherein the historical text data comprises historical record data and historical interaction data.

[0122] The intelligent labeling module is configured to aggregate the text data in the multi-modal data, the text data transcribed from the audio data, the text data transcribed from the video data, and the historical text data into total text data, and label the labelable text in the total text data.

[0123] The feature extraction module is configured to extract features of the total text data through a deep fusion model, wherein the deep fusion model is a fusion model of a text vectorization model and a multi-dimensional feature extraction model.

[0124] The text classification module is configured to calculate a text classification result.

[0125] The touch module is configured to reach the result.

[0126] It should be noted that details not disclosed in the refrigeration equipment system of the embodiments of the present application can refer to the details disclosed in the text classification method of the embodiments of the present application.

[0127] The refrigeration equipment system can further include a refrigeration equipment 100, a mobile phone, a computer, a notebook, a palm computer, a cloud server and other computing devices, and a processing module 50, a storage module 60, and a computer program stored in the storage module 60 and executable on the processing module 50, such as the text classification method program described above. The processing module 50 executes the computer program to implement the steps in each of the above text classification method embodiments, such as Figures 2 to 5 the steps shown.

[0128] The refrigeration equipment system can further include a signal transmission module and a communication bus 70. As Figure 7As shown, the signal transmission module is configured to transmit data to the processing module 50 or the server, for example, the data transmission between the refrigeration device 100 and the mobile phone, the data transmission between the refrigeration device 100 and the server through the signal transmission module. The signal transmission module can transmit data in the form of wireless connection, such as Bluetooth, wifi, ZigBee, etc. The communication bus 70 is configured to establish a connection between the signal transmission module, the processing module 50 and the storage module 60. The communication bus 70 can include a path for transmitting information between the above-mentioned signal transmission module, the processing module 50 and the storage module 60.

[0129] The processing module 50 and the storage module 60 can be integrated into a part of the refrigeration device 100, or a part of the mobile phone, a local terminal device, or a part of the cloud server.

[0130] The processing module 50 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or any conventional processor. The processing module 50 is the control center of the refrigeration device system, and connects various parts of the refrigeration device system through various interfaces and lines.

[0131] The storage module 60 can be configured to store the computer programs and / or modules. The processing module 50 realizes various functions of the refrigeration device system by running or executing the computer programs and / or modules stored in the storage module 60, and calling the data stored in the storage module 60. The storage module 60 can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function, etc. In addition, the storage module 60 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0132] For example, the computer program can be divided into one or more modules / units, which are stored in the storage module 60 and executed by the processing module 50 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the refrigeration equipment system.

[0133] Further, an embodiment of the present application provides a readable storage medium storing a computer program, which, when executed by the processing module 50, can implement the steps in the text classification method described above, that is, implement the steps in any one of the technical solutions in the text classification method described above.

[0134] The modules of the text classification method integrated in the form of software function units and sold or used as independent products can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. The computer program, when executed by the processing module 50, can implement the steps of each method embodiment described above.

[0135] The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, CD-ROM, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the contents included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0136] It should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description manner of the specification is only for the sake of clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that those skilled in the art can understand.

[0137] The above detailed description merely illustrates preferred and possible non-limiting implementations of the application, and is not intended to limit the scope of the application. Other equally effective embodiments and modifications will be apparent to those skilled in the art, and are intended to be included within the scope of the application.

Claims

1. A method of text classification, characterized by, The method comprises the following steps: transcribe all the multi-modal data and / or multi-source heterogeneous data into text data and aggregate the text data into total text data, wherein the multi-modal data comprises video data, and the transcription of the video data into text data comprises: obtaining key frame images in the video data through a diffusion network model; and recognizing the key frame images to generate text data; annotating the annotatable text in the total text data, and storing the annotated text as training data and test data respectively; extracting features of the total text data through a deep fusion network model, training the deep fusion network model using the training data, and predicting results through the deep fusion network model using the test data, wherein the deep fusion network model is a fusion model of a text vectorization model and a multi-dimensional feature extraction model, the text vectorization model vectorizes text data, and the multi-dimensional feature extraction model extracts multi-dimensional interaction features and correlation features; calculating a matching result of text classification based on the extracted features.

2. The text classification method of claim 1, wherein, The step of annotating the annotatable text in the total text data comprises: annotating the annotatable text in the total text data comprises:

3. The text classification method of claim 1, wherein, annotating the annotatable text in the total text data comprises: collecting multi-modal data and / or multi-source heterogeneous data and preprocessing, wherein the multi-modal data comprises text, audio and video data; transcribing the video data into text data; transcribing the audio data into text data; obtaining historical text data; aggregating the text data in the multi-modal data, the text data transcribed from the audio data, the text data transcribed from the video data, and the historical text data into the total text data.

4. The text classification method of claim 1, wherein, The text vectorization model encodes word, phrase, sentence level text.

5. The text classification method of claim 1, wherein, The multi-dimensional feature extraction model uses a multi-head attention mechanism to extract word, phrase, sentence interaction and correlation features, and context semantic information.

6. The text classification method of claim 1, wherein, The step of calculating a matching result of text classification based on the extracted features comprises: passing the vectors extracted from the features through a fully connected layer and a self-attention mechanism in sequence to obtain a text classification result.

7. The text classification method of claim 1, wherein, The method further comprises the following steps: reaching the calculated text classification result.

8. A method of text classification characterized by, The method comprises the following steps: transcribe all the multi-modal data and / or multi-source heterogeneous data into text data and aggregate the text data into total text data; cleaning data of the unannotatable text in the total text data; extracting features of the total text data through a deep fusion network model, training the deep fusion network model using the training data, and predicting results through the deep fusion network model using the test data, wherein the deep fusion network model is a fusion model of a text vectorization model and a multi-dimensional feature extraction model, the text vectorization model vectorizes text data, and the multi-dimensional feature extraction model extracts multi-dimensional interaction features and correlation features; calculating a matching result of text classification based on the extracted features. The step of annotating the annotatable text in the total text data comprises: annotating the annotatable text in the total text data comprises: annotating the annotatable text in the total text data comprises: collecting multi-modal data and / or multi-source heterogeneous data and preprocessing, wherein the multi-modal data comprises text, audio and video data; transcribing the video data into text data; transcribing the audio data into text data; obtaining historical text data; aggregating the text data in the multi-modal data, the text data transcribed from the audio data, the text data transcribed from the video data, and the historical text data into the total text data. The text vectorization model encodes word, phrase, sentence level text. The multi-dimensional feature extraction model uses a multi-head attention mechanism to extract word, phrase, sentence interaction and correlation features, and context semantic information. The step of calculating a matching result of text classification based on the extracted features comprises: passing the vectors extracted from the features through a fully connected layer and a self-attention mechanism in sequence to obtain a text classification result. The method further comprises the following steps: reaching the calculated text classification result.

9. The text classification method of claim 8, wherein, The step of transcribing the multi-modal data and / or multi-source heterogeneous data into text data and aggregating the text data into total text data comprises: collecting and preprocessing multi-modal data and / or multi-source heterogeneous data, wherein the multi-modal data comprises text, audio and video data; transcribing the video data into text data; transcribing the audio data into text data; obtaining historical text data; aggregating the text data in the multi-modal data, the text data transcribed from the audio data, the text data transcribed from the video data, and the historical text data into the total text data.

10. The text classification method of claim 9, wherein, The preprocessing comprises cleaning, format conversion and storage of the multi-modal data.

11. The text classification method of claim 9, wherein, The step of transcribing the video data into text data comprises: separating the audio and image in the video data to obtain audio data and image data; recognizing the text information in the image data to transcribe into text data; recognizing the image data based on spatio-temporal and long-distance dependence features to transcribe into text data.

12. The text classification method of claim 11, wherein, The step of recognizing the image based on spatio-temporal and long-distance dependence features to transcribe into text data comprises: generating a student model through a fusion model of knowledge distillation and diffusion model based on the recognition of the image based on spatio-temporal and long-distance dependence features, and transcribing the image data into text data through the student model.

13. The text classification method of claim 9, wherein, The step of transcribing the video data into text data comprises: obtaining key frame images in the video data through a diffusion network model; recognizing the key frame images to generate text data.

14. The text classification method of claim 9, wherein, The step of transcribing the audio data into text data comprises: combining the speech spatio-temporal characteristics and context relationship features in the data acquisition scenario to establish a deep recurrent convolutional network model based on the fusion neural network MMCNN-RNN, CTC and Attention to transcribe the audio data into text data.

15. The text classification method of claim 9, wherein, The historical text data comprises historical record data and historical interaction data, wherein the historical record data comprises user's food material preference data, interest data and comment data, and the historical interaction data comprises interaction records obtained from the client or the interaction end of the refrigeration equipment.

16. The text classification method of claim 8, wherein, The text vectorization model encodes the word, phrase, sentence level text.

17. The text classification method of claim 8, wherein, The multi-dimensional feature extraction model uses a multi-head attention mechanism to extract word, phrase, sentence interaction and correlation features, and context semantic information.

18. The text classification method of claim 8, wherein, The step of calculating the matching result of text classification based on the extracted features comprises: passing the vector after feature extraction through a full connection layer and a self-attention mechanism in sequence to obtain a text classification result.

19. The text classification method of claim 8, wherein, It also includes the step of: reaching the calculated text classification result.

20. A refrigeration appliance system characterized by, It includes: a storage module for storing a computer program; a processing module for executing the computer program to realize the steps in the text classification method of any one of claims 1 to 19.

Images