Text semantic matching method and refrigeration equipment system

Abstract

The application discloses a text semantic matching method and a refrigeration equipment system. The method comprises the following steps: labeling the markable text in the total text data; performing result matching on the unmarkable text and the labeled text in the total text data through slot extraction; outputting the matching result if the matching is successful; and calculating the matching result through a neural network if the matching fails. The matching of the text semantic matching method utilizes the mechanism of labeling first and then slot extraction. The slot extraction includes three cases: the unmarked text is matched successfully after slot extraction; the labeled text is matched successfully after slot extraction; and the labeled text is matched unsuccessfully after slot extraction. The deep fusion network model is used to calculate the matching of the type of matching failure. Therefore, the overall matching speed of the text is greatly improved, the matching accuracy is high, and the use experience of the user is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of refrigeration equipment, and in particular to a text semantic matching method and a refrigeration equipment system. BACKGROUND

[0002] With the progress of artificial intelligence, people hope to introduce artificial intelligence into the field of refrigerators to make refrigerators more intelligent. In the process of intelligentization of refrigerators, a large number of optimization processes for refrigerator scenes are involved, which include optimization of various types of interactions between users and refrigerators in spoken language, text, and video. In the optimization process, the inventors found that the prior art has the following problems:

[0003] The response speed of existing interactions is slow, and the accuracy is not high enough to meet the needs of users to communicate clearly and instantly, so that users feel that the human-computer dialogue is not as natural as human-to-human dialogue, and the use experience is poor. SUMMARY

[0004] To solve at least one of the above-mentioned problems in the prior art, the purpose of the present application is to provide a text semantic matching method and a refrigeration equipment system with fast interaction response speed and accurate feedback information.

[0005] To achieve the above-mentioned purpose of the application, an embodiment of the present application provides a text semantic matching method, comprising the following steps:

[0006] annotating the annotatable text in the total text data;

[0007] performing result matching on both the unannotatable text and the annotated text in the total text data through slot extraction, and judging the matching result;

[0008] outputting the matching result if the matching is successful;

[0009] if the matching fails, transmitting the text data corresponding to the failure to a deep fusion network model for feature extraction, and then calculating the matching result of the text semantic according to the feature extraction result; wherein the deep fusion network model is a fusion model of a text vectorization model and a multi-dimensional feature extraction model, the text vectorization model vectorizes the text data, and the multi-dimensional feature extraction model extracts multi-dimensional interaction features and correlation features.

[0010] As a further improvement of the present application, the step of performing result matching on both the unannotatable text and the annotated text in the total text data through slot extraction comprises:

[0011] performing result matching on both the unannotatable text and the annotated text in the total text data through a rule engine;

[0012] When the rule engine detects a problem, the rule definition is automatically analyzed and repaired by a quick repair module, and the result matching is re-performed by the rule engine.

[0013] As a further improvement of the present application, the steps further include:

[0014] When the quick repair module cannot solve the problem detected by the rule engine, or the rule engine still cannot perform result matching after the rule is repaired, the slot extraction matching fails; wherein the problem includes inaccurate rule definition, rule conflict, or low rule execution efficiency.

[0015] As a further improvement of the present application, the step of matching failure further includes:

[0016] The text data corresponding to the failure is transmitted to a deep fusion network model for feature extraction, and interactive features are extracted;

[0017] The aggregated interactive feature information between the interactive features is calculated;

[0018] The differential interactive feature information between the interactive features is calculated;

[0019] According to the feature information, the aggregated interactive feature information, and the differential interactive feature information, the matching result of the text semantics is calculated.

[0020] As a further improvement of the present application, the step of calculating the aggregated interactive feature information between the interactive features includes:

[0021] The aggregated interactive feature information between the interactive features is calculated by attention weighted summation;

[0022] The step of calculating the differential interactive feature information between the interactive features includes:

[0023] The differential interactive feature information between the interactive features is calculated by attention mechanism enhancement.

[0024] As a further improvement of the present application, the step of labeling the labelable text in the total text data includes:

[0025] The labelable text is sequentially pre-labeled, formally labeled, and labeled quality inspected, wherein when the value of the text after formal labeling is lower than the preset threshold value after quality inspection, the text is returned to the pre-labeling for re-labeling.

[0026] As a further improvement of the present application, the step of labeling the labelable text in the total text data includes:

[0027] The labelable text in the total text data is labeled, and the labeled text is stored as training data and test data, respectively;

[0028] the step of transmitting the corresponding text data of the failed matching to the deep fusion network model for feature extraction comprises:

[0029] for the labeled text data of the failed matching, training the deep fusion network model with the training data, and predicting the result through the deep fusion network model with the test data.

[0030] as a further improvement of the present application, further comprising the steps of:

[0031] data cleaning on the unannotated text in the total text data;

[0032] the step of transmitting the corresponding text data of the failed matching to the deep fusion network model for feature extraction comprises:

[0033] for the unannotated text data of the failed matching, training the deep fusion network model with the unannotated text by using unsupervised learning algorithm.

[0034] as a further improvement of the present application, the total text data is obtained by converting all the multi-modal data and / or multi-source heterogeneous data into text data and then aggregating them.

[0035] as a further improvement of the present application, further comprising the steps of:

[0036] collecting multi-modal data and / or multi-source heterogeneous data and preprocessing, wherein the multi-modal data includes text, audio and video data, and the preprocessing includes cleaning, format conversion and storage of the multi-modal data;

[0037] transcribing the video data into text data;

[0038] transcribing the audio data into text data;

[0039] obtaining historical text data;

[0040] aggregating the text data in the multi-modal data, the text data transcribed from the audio data, the text data transcribed from the video data, and the historical text data into the total text data.

[0041] as a further improvement of the present application, the step of transcribing the video data into text data comprises:

[0042] separating the audio and image in the video data to obtain audio data and image data;

[0043] recognizing the text information in the image data and transcribing it into text data;

[0044] The image data is recognized based on space-time and long-distance dependence features, and is transcribed into text data.

[0045] The step of recognizing the image based on space-time and long-distance dependence features and transcribing the image into text data comprises:

[0046] The student model is generated by fusing the knowledge distillation and diffusion model based on the space-time and long-distance dependence features of the recognized image, and the image data is transcribed into text data by the student model.

[0047] As a further improvement of the present application, the step of transcribing the video data into text data comprises:

[0048] The key frame image in the video data is obtained by a diffusion network model.

[0049] The key frame image is recognized to generate text data.

[0050] As a further improvement of the present application, the step of transcribing the audio data into text data comprises:

[0051] The voice space-time characteristics and context relationship features in the combined data acquisition scene are combined to establish a deep recurrent convolutional network model based on the fusion neural network MMCNN-RNN, CTC, and Attention, and the audio data is transcribed into text data.

[0052] As a further improvement of the present application, the historical text data comprises historical record data and historical interaction data, wherein the historical record data comprises user's food material preference data, interest data and comment data, and the historical interaction data comprises interaction records obtained from the interaction end of the client or the refrigeration equipment.

[0053] As a further improvement of the present application, the text vectorization model encodes the word, phrase, sentence level text.

[0054] The multi-dimensional feature extraction model uses a multi-head attention mechanism to extract word, phrase, sentence interaction and correlation features, and context semantic information.

[0055] As a further improvement of the present application, the step of calculating the matching result of the text semantic according to the feature extraction result comprises: the vectors after feature extraction are sequentially subjected to full connection layer and self-attention mechanism to obtain the text semantic matching result.

[0056] As a further improvement of the present application, it further comprises the steps of:

[0057] The matching results of the matching success and the matching failure after re-computation are reached.

[0058] To achieve the above-mentioned one of the purposes of the application, an embodiment of the application provides a refrigeration equipment system, comprising:

[0059] A storage module is configured to store the computer program.

[0060] A processing module is configured to implement the steps of the text semantic matching method when executing the computer program.

[0061] Compared with the prior art, the text semantic matching method has the following beneficial effects: the matching of the text semantic matching method utilizes the mechanism of first labeling and then slot extraction, so that some text data is easier to be extracted after being labeled, and the unlabeled data is generally a phrase, and the phrase is easy to be extracted, so that during the slot extraction, three cases are included: the unlabeled data is successfully matched after being extracted; the labeled data is successfully matched after being extracted; the labeled data is unsuccessfully matched after being extracted, and the data of the type of the unsuccessful matching is matched again through the deep fusion network model and calculation; the overall matching speed of the text is greatly improved, the matching accuracy is high, and the use experience of the user is improved. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is a structural schematic diagram of the refrigeration equipment system of an embodiment of the application;

[0063] Figure 2 is a flowchart of a structural schematic diagram of the text semantic matching method of an embodiment of the application;

[0064] Figure 3 is a partial flowchart of a structural schematic diagram of the text semantic matching method of an embodiment of the application;

[0065] Figure 4 is a flowchart of slot extraction of an embodiment of the application;

[0066] Figure 5 is a data flowchart of the text semantic matching method of an embodiment of the application;

[0067] Figure 6 is a module schematic diagram of an embodiment of the application;

[0068] Figure 7 is a structural block diagram of the refrigeration equipment system of an embodiment of the application;

[0069] Among them, 100, refrigeration equipment; 10, interactive screen; 20, camera; 30, microphone; 40, loudspeaker; 50, processing module; 60, storage module; 70, communication bus; 200, client. DETAILED DESCRIPTION

[0070] The present invention will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by those skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0071] An embodiment of the present invention provides a text semantic matching method and a refrigeration equipment system with fast interactive response speed and accurate feedback information.

[0072] The refrigeration equipment system may include a refrigeration equipment 100 and a client 200 corresponding to the refrigeration equipment 100. The refrigeration equipment 100 may be a refrigerator, and the client 200 may be a mobile phone or an app on the mobile phone. Figure 1 As shown, the refrigeration device 100 and the client 200 can be connected via wireless signals. The refrigeration device 100 is described below using a refrigerator as an example.

[0073] Continue as Figure 1 As shown, the refrigeration device 100 can be a refrigerator with audio acquisition, video acquisition, and interaction with a user interface. The refrigerator is provided with a microphone 30 for collecting audio, a camera 20 for shooting video, a speaker 40 for interacting with the user's voice, and an interactive screen 10 for interacting with the user's text or graphic interface. The interactive screen 10 can be set on the door of the refrigerator; after the user opens the refrigerator door, the camera 20 records the user's operation and forms video data; the speaker 40 and the microphone 30 are combined to interact with the user in the form of question and answer audio.

[0074] Taking a mobile phone as an example, the client 200 can communicate with the refrigerator via text or voice through the mobile phone, or manage the food information in the refrigerator and control the operating status of the refrigerator through the mobile phone.

[0075] In addition, the refrigeration equipment system may also include other external devices, such as external temperature sensors, cameras 20 or microphones 30 and speakers 40 provided by other devices, smart speakers, etc. These devices can be connected to the refrigeration equipment 100 or the client 200 via wireless signals.

[0076] Data from various devices within the refrigeration system forms multi-source heterogeneous data. This data, collected by multiple devices, can be transmitted via wired, Wi-Fi, Bluetooth, and other means. Text, audio, video, and other types of data constitute multimodal data. This data can be real-time, online, offline, or stored historical data.

[0077] The refrigeration equipment scene includes the interaction of the user with the refrigeration equipment 100, the interaction of the user with the corresponding client 200 of the refrigeration equipment 100, for example, the interaction of the food, the interaction of the instruction, the recording of the video when the user operates the refrigeration equipment 100, the control of the user on the internal temperature and humidity of the refrigeration equipment 100, and the review of the user on the food on the client 200, the preference of the user and the like, the data generated by the direct operation of the refrigeration equipment 100 and the data generated by the client 200 related to the refrigeration equipment 100 are all the data in the refrigeration equipment scene.

[0078] The embodiment fully mines the semantics, syntax and context information of natural language understanding of the data itself and the data by using the multi-modal real-time and offline data generated in the use scene of the refrigeration equipment 100 and the accumulated mass text historical data, so that the text semantic matching result in the refrigeration equipment scene is more accurate.

[0079] The core idea of semantic matching is to convert text into semantic vector representation and calculate the distance or similarity of other vectors close to these vectors to judge the relevance between texts. Semantic matching can help us better understand and process natural language text. It can be applied to various scenarios, such as text classification, knowledge graph construction, intelligent customer service, etc.

[0080] Generally, semantic matching needs to compare the similarity between a large number of texts, and the calculation speed is slow, and the matching needs a long time. However, the present application greatly accelerates the matching speed by pre-annotation and slot extraction. In slot extraction, there are three cases: (1) unannotated and successfully matched after slot extraction; (2) annotated and successfully matched after slot extraction; (3) annotated and failed to match after slot extraction. For the type that fails to match, a deep fusion network model and calculation matching are used. That is, the matching speed of two types of documents is improved, and the remaining type that fails to match is calculated by a neural network, which greatly improves the speed of the existing semantic matching, and the slot extraction and neural network matching accuracy are also very high. Therefore, the present application greatly accelerates the semantic matching speed and improves the accuracy. The matching method is further described below.

[0081] The following describes an embodiment of the present application in combination with Figure 2 and Figure 5 , although the present application provides the method operation steps as shown in the following embodiments or flowcharts, based on conventional or non-creative labor, there is no necessary causal relationship between the steps in the logicality of the method, and the execution order of these steps is not limited to the execution order provided in the embodiments of the present application. For example, the acquisition order of the following steps S20, S30 and S40 can be adjusted arbitrarily or simultaneously, and the time sequence is not distinguished.

[0082] Step S10: Collecting multi-modal and / or multi-source heterogeneous data and preprocessing, wherein the multi-modal data includes text, audio and video data related to the refrigeration equipment scene.

[0083] The preprocessing includes cleaning, format conversion and storage of the multi-modal data. The format conversion includes parsing of the data format.

[0084] Text is collected through the interaction screen 10 of the client 200 and / or the refrigerator end. The client 200 can include mobile phones, pads, PC ends and other devices. In addition to the text data generated on the application program, it also includes customer service text, web text, and text on the applet or public number. The preprocessing of the text data can use stop words, de-duplication, etc.

[0085] Audio data can be collected by the microphone 30 of the mobile phone end and / or the refrigerator end, or can be connected to other sound collecting units through WiFi or other wireless connections to obtain the voice of the sound collecting unit on other devices. The microphone 30 can be a single microphone 30 or an array of microphones 30.

[0086] Video data can be collected by the camera 20 of the client 200 and / or the refrigerator end, such as through the app of the mobile phone, the camera 20, Bluetooth, etc. In addition, the voice and video can be separated by a script to obtain effective voice and video data.

[0087] The above-mentioned various ways complete the multi-channel and terminal multi-modal data collection task, ensuring data integrity and multi-modal cognitive characteristics.

[0088] Most of the prior art only uses text as a single data, which leads to the neglect of other data in the later data learning, and further reduces the recognition accuracy when the user interacts with the refrigerator in other forms. The text source of the present application includes various modal data generated by the user, which is further trained and matched with the text semantic, and has higher accuracy.

[0089] Step S20: Transcribing the video data into text data.

[0090] This step includes the following two embodiments. In one embodiment, it includes the following steps:

[0091] Separating the audio and image in the video data to obtain audio data and image data;

[0092] Recognizing the text information in the image data and transcribing it into text data;

[0093] Recognizing the image data based on space-time and long-distance dependence features and transcribing it into text data.

[0094] The recognition of the text in the image data can directly transcribe it into text data, and the recognition of the non-text in the image can be the recognition of the content said by the user according to the changes of the multiple images of the user's mouth movement. And the application considers that if only the image features of the speaker are used, the recognized sentence may be complex, so the application combines the sentence length factor and the context relevance for recognition, the sentence length factor includes the features of different sentence lengths and different word compositions, and the recognition is performed based on the space-time and long-distance dependence features to mine rich semantic feature information of the sentence sequence.

[0095] In addition, in order to speed up the response speed of the model, a student model is generated by fusing a knowledge distillation model and a diffusion model based on the space-time and long-distance dependence feature recognition image, the knowledge of the original large model is migrated to the student network, and the image data is transcribed into text data by the student model. The student model is a model with discrete time steps and short steps, and the student model can be distilled to half of the steps of the teacher model.

[0096] In another embodiment, the key frame image in the video data can be obtained by a diffusion model;

[0097] The key frame image is recognized to generate text data.

[0098] Here, the content of the image is explained in the form of text by recognizing the meaning of the image, and the transcription of the image data into text data is completed.

[0099] Step S30: The audio data is transcribed into text data by combining the speech space-time characteristics and the context relationship features of the refrigeration equipment 100.

[0100] The audio data of this step can be directly collected audio data, and can also include audio data segmented from video data.

[0101] This embodiment combines the speech data space-time characteristics of the refrigeration equipment 100 such as a refrigerator, and establishes a deep recurrent convolutional network model based on a fusion neural network MMCNN-RNN, CTC, and Attention through an end-to-end learning method to transcribe the audio data into text data, so as to obtain rich high-level speech feature information and improve the accuracy of the model in transcribing speech into text.

[0102] Step S40: Obtain historical text data, wherein the historical text data includes historical record data and historical interaction data.

[0103] Compared with the historical text data, the multi-modal data collected in the above step S10 is real-time data, and the historical data obtained in the step S40 is historical data. By obtaining the historical text data, on the one hand, the information of the historical text data itself can be utilized, and on the other hand, the historical text data can be complementary and associated with the real-time data, so as to fully obtain the semantic information of the text data.

[0104] The historical text data can be a lot of unlabeled texts accumulated on the refrigeration equipment 100 or the client 200. After obtaining the historical text data, the collected data can be uniformly processed, such as cleaning and format conversion, so that the text formats of the historical text data and the real-time collected data are uniform, and the comprehensiveness and particularity of the data characteristics are ensured.

[0105] Further, the historical record data includes user's food material preference data, interest data and comment data; and the historical interaction data includes interaction records obtained from the interaction end of the client 200 or the refrigeration equipment 100. The historical text data also includes data collected by the refrigeration equipment 100 and data on the client 200 corresponding to the refrigeration equipment 100.

[0106] Step S50: preprocessing the total text data.

[0107] The text data in the multi-modal data, the text data converted from the audio data, the text data converted from the video data, and the historical text data are summarized as total text data.

[0108] The summary as total text data reflects that the embodiment adopts multi-source heterogeneous data, real-time, offline voice, video, image, text and user historical comments, food material preference, food material interest and other text data.

[0109] The total text data can include two types of data, the first type is data that can be pre-annotated, and the second type is data that cannot or does not need to be annotated. The data that cannot or does not need to be annotated is generally a phrase, and the phrase is easy to extract slot in the subsequent steps. The data that needs to be pre-annotated is generally a long sentence. The annotation of the sentence can facilitate subsequent classification and recognition, and the annotated data is more conducive to matching the result in the slot extraction in the subsequent steps.

[0110] For the annotatable text data, the following steps can be used:

[0111] Step S51: annotating the annotatable text in the total text data, and storing the annotated text as training data and test data respectively.

[0112] Specifically, as Figure 5As shown, the markable text is sequentially pre-labeled, formally labeled, labeled quality inspection and data storage, wherein when the value of the text after formal labeling is lower than the preset threshold after labeled quality inspection judgment, the text is returned to pre-labeling for re-labeling. The labeled data can include training data and test data.

[0113] For text data that cannot be labeled or does not need to be labeled, the following steps can be taken:

[0114] Step S52: Data cleaning for text data that cannot be labeled in the total text data.

[0115] In step S52, these text data that cannot be labeled are directly cleaned, formatted, and other tasks, and then directly participate in subsequent slot extraction without labeling.

[0116] Step S60: Perform result matching on both the text data that cannot be labeled and the labeled text data through slot extraction, and judge the matching result, such as Figure 3 As shown.

[0117] If the matching is successful, go to the subsequent step S80;

[0118] If the matching fails, proceed to the subsequent step S70.

[0119] Further, the process of slot extraction is as shown in Figure 4 The rule engine is used to perform result matching on both the text data that cannot be labeled and the labeled text data.

[0120] When the rule engine detects a problem, the quick fix module automatically analyzes and fixes the rule definition, and re-performs result matching through the rule engine.

[0121] A rule engine is a software tool that can define and execute various rules in a system, such as business rules, process rules, data validation rules, etc. Rule engines may have various problems, such as inaccurate rule definitions, rule conflicts, low rule execution efficiency, etc. When the rule engine detects a violation of the rules, it can generate a warning or error message.

[0122] Quick fix technology can quickly identify and solve many problems. Quick fix technology usually involves automated testing, code analysis, debugging tools, etc. It can quickly identify problems and provide solutions.

[0123] Therefore, the embodiment realizes the rapid repair of rules by combining the rule engine and the rapid repair technology. When the rule engine finds a problem, the rapid repair technology can automatically analyze and repair the rule definition, reduce the time and cost of manual repair, improve the reliability and efficiency of the system, realize the effect of real-time update and repair of rules, and avoid the problem of offline rules without update, which requires a lot of time and effort to define and maintain the rule set.

[0124] When the rapid repair module cannot solve the problem detected by the rule engine, or the rule engine still cannot perform result matching after the rule is repaired, the slot extraction matching fails.

[0125] The following is the semantic matching of the text of the slot extraction failure.

[0126] Step S70: calculating the semantic matching result through the neural network.

[0127] Step S70, as shown in Figure 3 , specifically includes steps S71, S72, and S73.

[0128] Step S71: performing feature extraction on the total text data through a deep fusion network model, training the deep fusion network model with the training data, and predicting the result through the deep fusion network model with the test data.

[0129] The labeled data of step S51 in the foregoing can include training data and test data. Step S71 can perform a pre-training task on the constructed deep fusion model based on the training data, and then perform result prediction based on the test data to obtain rich semantic feature information, so as to obtain the best effective model, thereby ensuring the optimal prediction result and higher accuracy of user feedback information.

[0130] The text vectorization model of step S71 encodes the word, word, phrase, and sentence level text; the multi-dimensional feature extraction model uses a multi-head attention mechanism to extract word, word, phrase, sentence interaction and correlation features, and context semantic information.

[0131] Step S72: transmitting the text data corresponding to the failure to the deep fusion network model for feature extraction to extract interaction features.

[0132] The aggregation interaction feature information between the interaction features is calculated by attention weighted summation.

[0133] The differential interaction feature information between the interaction features is calculated by attention mechanism enhancement.

[0134] The aggregated interaction feature information can aggregate the interaction information between different features, thereby obtaining richer feature representation. It can improve the prediction performance of the model, especially in the case of complex interaction between features, by learning and aggregating feature information through attention weighted summation to obtain richer feature representation.

[0135] The differentiated interaction feature information is constructed by calculating the difference between different features. It can capture important interaction relationships between different features, thereby improving the prediction ability of the model. The calculation method of the differentiated interaction feature information can be to enhance the calculation of the difference between the interaction features by using the attention mechanism, and the differentiated interaction feature information can be used as the input of the model, and the subsequent neural network model is learned and aggregated to obtain more accurate prediction results.

[0136] Step S73: calculating the matching result of the text semantic according to the feature information, the aggregated interaction feature information and the differentiated interaction feature information.

[0137] In step S73, the vector after feature extraction is sequentially calculated through the full connection layer and the self-attention mechanism to obtain the semantic matching result of the text. Here, the semantic matching result is first calculated through the full connection layer, and then the semantic relationship quantization result with stronger interaction between texts is calculated through the self-attention mechanism.

[0138] In addition, step S73 can also use the semantic matching similarity with elimination to calculate, which is realized based on the threshold size control of the distance between the word items and the word items. In this way, some redundant information in the text can be removed.

[0139] Step S80: result reaching.

[0140] The semantic matching result after the above steps is run is used for reaching task, and the result reaching method can adopt various forms with or without external, such as external call, short message reaching, email notification, large screen display, voice broadcast, text output, smart speaker, pop-up window UI, app end, PAD, web end and various result reaching modes, which meets the demand of result display mode and digital display.

[0141] Compared with the prior art, the embodiment has the following beneficial effects:

[0142] The matching of this text semantic matching method utilizes the mechanism of first labeling and then slot extraction, so that some text data are easier to slot extract after being labeled, and the unlabeled ones are generally phrases, which are easy to slot extract. Therefore, when extracting the slots, there are three situations: the unlabeled ones are successfully matched after slot extraction; the labeled ones are successfully matched after slot extraction; the labeled ones fail to match after slot extraction, and the category of failed matching is then matched through a deep fusion network model and calculation; the overall matching speed of the text is greatly improved, and the matching accuracy is high, which improves the user experience.

[0143] In one embodiment, the present invention also proposes a refrigeration equipment system, which includes a storage module 60 and a processing module 50. When the processing module 50 executes the computer program, it can implement the steps in the above-mentioned text semantic matching method, that is, implement the steps in any one of the technical solutions in the above-mentioned text semantic matching method.

[0144] Refrigeration equipment system can also be Figure 6 As shown, it includes the following modules, and the specific functions of each module are as follows:

[0145] An acquisition module, configured to acquire and preprocess multimodal data and / or multi-source heterogeneous data, wherein the multimodal data includes text, audio, and video data related to the refrigeration equipment scenario;

[0146] A video transcription module, used to transcribe video data into text data;

[0147] An audio transcription module, configured to transcribe audio data into text data by combining the spatiotemporal characteristics of speech and contextual features of the refrigeration appliance 100;

[0148] An acquisition module, configured to acquire historical text data, wherein the historical text data includes historical record data and historical interaction data;

[0149] an intelligent annotation module for aggregating the text data in the multimodal data, the text data transcribed from the audio data, the text data transcribed from the video data, and the historical text data into total text data, and annotating the annotatable text in the total text data;

[0150] A slot extraction module, configured to perform result matching on both the unlabeled text and the labeled text in the total text data through slot extraction;

[0151] A feature extraction module, configured to extract features from the total text data through a deep fusion model after matching fails, wherein the deep fusion model is a fusion model of a text vectorization model and a multi-dimensional feature extraction model;

[0152] An aggregation difference module is configured to calculate aggregated interaction feature information between interaction features and calculate differentiated interaction feature information between interaction features.

[0153] A semantic matching module is configured to calculate a semantic matching result.

[0154] A reach module is configured to reach a result.

[0155] It should be noted that details of the refrigeration equipment system in the embodiments of the present application are not disclosed, and please refer to the details disclosed in the text semantic matching method in the embodiments of the present application.

[0156] The refrigeration equipment system can further include a refrigeration equipment 100, a mobile phone, a computer, a notebook, a palm computer, a cloud server and other computing devices, and include but not limited to a processing module 50, a storage module 60, and a computer program stored in the storage module 60 and executable on the processing module 50, such as the text semantic matching method program described above. The processing module 50 executes the computer program to realize the steps in each of the above text semantic matching method embodiments, such as Figures 2 to 5 the steps shown.

[0157] The refrigeration equipment system can further include a signal transmission module and a communication bus 70. As Figure 7 shown, the signal transmission module is configured to send data to the processing module 50 or the server, for example, the refrigeration equipment 100 and the mobile phone, the refrigeration equipment 100 and the server transmit data through the signal transmission module, the signal transmission module can transmit data in the form of wireless connection, such as Bluetooth, wifi, ZigBee, etc., the communication bus 70 is configured to establish a connection between the signal transmission module, the processing module 50 and the storage module 60, the communication bus 70 can include a channel to transmit information between the above signal transmission module, the processing module 50 and the storage module 60.

[0158] The processing module 50 and the storage module 60 can be integrated into a part of the refrigeration equipment 100, or a part of the mobile phone, a local terminal device, or a part of the cloud server.

[0159] The processing module 50 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processing module 50 is the control center of the refrigeration equipment system, and connects various parts of the refrigeration equipment system through various interfaces and lines.

[0160] The storage module 60 can be used to store the computer programs and / or modules, and the processing module 50 realizes various functions of the refrigeration equipment system by running or executing the computer programs and / or modules stored in the storage module 60 and calling the data stored in the storage module 60. The storage module 60 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function, etc. In addition, the storage module 60 can include a high-speed random access memory, and can also include a non-volatile memory such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0161] For example, the computer program can be divided into one or more modules / units, which are stored in the storage module 60 and executed by the processing module 50 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the refrigeration equipment system.

[0162] Further, an embodiment of the present application provides a readable storage medium storing a computer program, which is executed by the processing module 50 to realize the steps in the above-mentioned text semantic matching method, that is, to realize the steps in any one of the technical solutions in the above-mentioned text semantic matching method.

[0163] The text semantic matching method integrated module, if in the form of software function unit is realized and is sold or used as independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium. The computer program is executed by the processing module 50, and the steps of each method embodiment can be realized.

[0164] The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, floppy disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0165] It should be understood that although the present specification is described in terms of embodiments, not every embodiment contains only one independent technical solution, and the description of the specification is only for the sake of clarity. The skilled person should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that the skilled person can understand.

[0166] The above series of detailed descriptions are only specific descriptions of the feasible embodiments of the present application, and are not intended to limit the protection scope of the present application. Any equivalent embodiments or changes made without departing from the spirit of the present application should be included in the protection scope of the present application.

Claims

1. A method for text semantic matching, characterized in that, The method comprises the following steps: annotating the annotatable text in the total text data; performing result matching on the unannotatable text and the annotated text in the total text data through slot extraction, and judging the matching result; if the matching is successful, outputting the matching result; if the matching fails, transmitting the text data corresponding to the failure to a deep fusion network model for feature extraction, extracting interactive features, calculating the aggregated interactive feature information between the interactive features through attention weighted summation, calculating the differential interactive feature information between the interactive features through attention mechanism enhancement, and calculating the matching result of the text semantics through full connection layer and self-attention mechanism according to the feature information, the aggregated interactive feature information, and the differential interactive feature information; wherein the deep fusion network model is a fusion model of a text vectorization model and a multi-dimensional feature extraction model, the text vectorization model vectorizes the text data, the text vectorization model encodes the word, phrase, sentence level text, and the multi-dimensional feature extraction model uses a multi-head attention mechanism to extract word, phrase, sentence interactive and correlation features, and context semantic information.

2. The text semantic matching method according to claim 1, characterized in that, The step of performing result matching on the unannotatable text and the annotated text in the total text data through slot extraction comprises: performing result matching on the unannotatable text and the annotated text in the total text data through a rule engine; when the rule engine detects a problem, automatically analyzing and repairing the rule definition through a quick repair module, and re-performing result matching through the rule engine.

3. The text semantic matching method according to claim 2, characterized in that, The method further comprises the following steps: when the quick repair module cannot solve the problem detected by the rule engine, or the rule engine still cannot perform result matching after the rule is repaired, the slot extraction matching fails; wherein the problem includes inaccurate rule definition, rule conflict, or low rule execution efficiency.

4. The text semantic matching method of claim 1, wherein, The step of annotating the annotatable text in the total text data comprises: annotating the annotatable text in the total text data, and storing the annotated text as training data and testing data respectively; 5. The method of claim 1, wherein, The step of transmitting the text data corresponding to the failure to a deep fusion network model for feature extraction comprises: for the annotated text data that fails to match, training the deep fusion network model using the training data, and predicting the result through the deep fusion network model using the testing data. The method further comprises the following steps: performing data cleaning on the unannotatable text in the total text data; 6. The method of claim 1, wherein, The step of transmitting the text data corresponding to the failure to a deep fusion network model for feature extraction comprises: for the unannotatable text data that fails to match, training the deep fusion network model using the unannotatable text through an unsupervised learning algorithm. ​ ​ 7. The text semantic matching method of claim 1, wherein, The total text data is obtained by transcribing all the multi-modal data and / or multi-source heterogeneous data into text data and aggregating them.

8. The text semantic matching method according to claim 7, characterized in that, Further comprising steps: collecting and preprocessing multi-modal data and / or multi-source heterogeneous data, wherein the multi-modal data includes text, audio and video data, and the preprocessing includes cleaning, format conversion and storage of the multi-modal data; transcribing video data into text data; transcribing audio data into text data; acquiring historical text data; aggregating the text data in the multi-modal data, the transcribed text data of the audio data, the transcribed text data of the video data, and the historical text data into the total text data.

9. The text semantic matching method of claim 8, wherein, The step of transcribing video data into text data includes: separating audio and image in the video data to obtain audio data and image data; recognizing text information in the image data and transcribing it into text data; recognizing image data based on spatio-temporal and long-distance dependence features and transcribing it into text data; The step of recognizing image based on spatio-temporal and long-distance dependence features and transcribing it into text data includes: generating a student model by a fusion model of knowledge distillation and diffusion model based on the recognition of image based on spatio-temporal and long-distance dependence features, and transcribing image data into text data through the student model.

10. The text semantic matching method of claim 8, wherein, The step of transcribing video data into text data includes: obtaining key frame images in the video data through a diffusion network model; recognizing the key frame images to generate text data.

11. The text semantic matching method of claim 8, wherein, The step of transcribing audio data into text data includes: combining the speech spatio-temporal characteristics and context relationship features in the scene of acquiring data, and establishing a deep recurrent convolutional network model based on the fusion neural network MMCNN-RNN, CTC and Attention to transcribe audio data into text data.

12. The text semantic matching method of claim 8, wherein, The historical text data includes historical record data and historical interaction data, wherein the historical record data includes user's food material preference data, interest data and comment data, and the historical interaction data includes interaction records obtained from the client or the interaction end of the refrigeration equipment.

13. The method of claim 1, wherein, Further comprising steps: reaching the matching results of the matching success and the matching results of the matching failure after recalculation.

14. A refrigeration appliance system characterized by, including: a storage module for storing a computer program; a processing module for executing the computer program to realize the steps in the text semantic matching method of any one of claims 1 to 13.

Images