Interaction method and device of refrigeration equipment, refrigeration equipment and electronic equipment

Abstract

The invention discloses an interaction method and device of refrigeration equipment, the refrigeration equipment and electronic equipment, and belongs to the technical field of refrigeration equipment. The method comprises the following steps: acquiring interaction information; under the condition that the interaction information comprises the audio information, extracting rhythm characteristics of the interaction information; determining a plurality of context features of different dimensions of the audio information based on the rhythm features; based on the plurality of context features and the weight parameters corresponding to the plurality of context features, obtaining semantic features of the audio information; and inputting the semantic features of the audio information into a pre-trained language model, so that the language model outputs an interaction result corresponding to the interaction information. The interaction result obtained by the method is high in accuracy, high in response speed and good in user experience.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of refrigeration equipment, and particularly relates to an interaction method and device of a refrigeration equipment, the refrigeration equipment and an electronic device. BACKGROUND

[0002] An intelligent refrigerator is usually equipped with an interaction system, which can interact with a user through voice or other manners to realize intelligent functions such as food material shelf life query, recipe recommendation or health problem consultation.

[0003] However, the accuracy of language recognition and interaction of the current intelligent refrigerator for a user is insufficient, and in actual use, interaction delay or even misrecognition problems may occur, affecting the operation experience of the user. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides an interaction method and device of a refrigeration equipment, the refrigeration equipment and an electronic device, which have high accuracy of interaction results, fast response speed and good user experience.

[0005] In a first aspect, the present application provides an interaction method of a refrigeration equipment, which comprises: obtaining interaction information; in the case where the interaction information comprises audio information, extracting prosodic features of the interaction information; determining a plurality of context features of different dimensions of the audio information based on the prosodic features; obtaining a semantic feature of the audio information based on the plurality of context features and weight parameters corresponding to the plurality of context features; inputting the semantic feature of the audio information into a pre-trained language model to make the language model output an interaction result corresponding to the interaction information.

[0006] According to the interaction method of the refrigeration equipment provided by the embodiments of the present application, the prosodic features can represent the speech rhythm, intonation and emotional expression manner in the audio information, and by extracting the prosodic features in the audio information, the content of the audio information can be more accurately captured and recognized. The interaction result obtained based on the prosodic features has high accuracy and fast response speed, and the user experience is good.

[0007] According to one embodiment of the present application, the determining of the plurality of context features of different dimensions of the audio information based on the prosodic features comprises: The prosodic features are input into a first processing model to obtain the multiple contextual features output by the first processing model. The first processing model includes multiple attention modules, which are used to obtain the contextual features of the prosodic features based on a bidirectional attention mechanism. The contextual features obtained by different attention modules have different dimensions.

[0008] According to one embodiment of this application, the weight parameters corresponding to the plurality of context features are obtained through the following steps: The multiple context features are input into the second processing model to obtain the weight parameters corresponding to the multiple context features output by the second processing model. The second processing model includes at least a first gated loop unit and a second gated loop unit. The first gated loop unit and the second gated loop unit are used to process the multiple context features based on time order. The processing order of the first gated loop unit and the second gated loop unit is opposite.

[0009] According to one embodiment of this application, the interactive information further includes text information, and the method further includes: The text information is subjected to text enhancement processing to obtain a first processing result, wherein the text enhancement processing includes at least one of noise processing and sparsity processing; The semantic features of the text information are obtained by extracting features from the first processing result. The semantic features of the text information are input into a pre-trained language model so that the language model outputs the interaction result corresponding to the interaction information.

[0010] According to one embodiment of this application, the interactive information further includes image information, and the method further includes: The image information is encoded to obtain a second processing result; Based on the second processing result, determine the vectorized representation of the second processing result; Based on the vectorized representation, the semantic features of the image information are determined; The semantic features of the image information are input into a pre-trained language model so that the language model outputs the interaction result corresponding to the interaction information.

[0011] According to one embodiment of this application, the prosodic features include at least one of intonation features, stress features, rhythm features, scale features, pitch features, and boundary dimension features.

[0012] According to one embodiment of this application, the language model includes a first sub-model and a second sub-model, wherein the response speed of the first sub-model is greater than that of the second sub-model, and the reasoning ability of the second sub-model is stronger than that of the first sub-model. The step of inputting the semantic features of the audio information into a pre-trained language model, so that the language model outputs the interaction result corresponding to the interaction information, includes: If the complexity of the audio information is less than a preset complexity threshold, the interaction result is inferred through the first sub-model; Alternatively, if the complexity of the audio information is greater than or equal to a preset complexity threshold, the interaction result can be inferred through the second sub-model.

[0013] According to one embodiment of this application, the interaction result includes at least one of voice broadcasting, text display, and image display.

[0014] Secondly, this application provides an interaction device for a refrigeration device, the device comprising: The acquisition module is used to acquire interactive information; The first processing module is used to extract the prosodic features of the interactive information when the interactive information includes audio information; The second processing module is used to determine multiple contextual features of the audio information in different dimensions based on the prosodic features; The third processing module is used to obtain the semantic features of the audio information based on the multiple context features and the weight parameters corresponding to the multiple context features; The fourth processing module is used to input the semantic features of the audio information into a pre-trained language model so that the language model outputs the interaction result corresponding to the interaction information.

[0015] According to the interactive device of the cooling equipment provided in the embodiments of this application, prosodic features can characterize the speech rhythm, intonation and emotional expression in audio information. By extracting prosodic features from audio information, the content of audio information can be captured and identified more accurately. The interactive results obtained based on prosodic features have high accuracy, fast response speed and good user experience.

[0016] Thirdly, this application provides a refrigeration device, which includes: The interactive device of the refrigeration equipment as described in the second aspect above.

[0017] According to the refrigeration device provided in the embodiments of this application, prosodic features can characterize the speech rhythm, intonation and emotional expression in audio information. By extracting prosodic features from audio information, the content of audio information can be captured and identified more accurately. The interaction results obtained based on prosodic features have high accuracy, fast response speed and good user experience.

[0018] Fourthly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the interaction method of the cooling device as described in the first aspect above.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts illustrating the interaction method of the refrigeration equipment provided in the embodiments of this application; Figure 2 This is a second schematic flowchart of the interaction method of the refrigeration equipment provided in the embodiments of this application; Figure 3 This is the third flowchart illustrating the interaction method of the refrigeration equipment provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the refrigeration equipment provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the interactive device of the refrigeration equipment provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0022] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0023] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with authorization from the owner of the relevant device.

[0024] It should be noted that the refrigeration equipment in this embodiment can be understood as a broad refrigeration storage device, including but not limited to refrigerators, freezers, display cases, beverage cabinets, wine cabinets, refrigerated display cases, and refrigerated vending machines. Refrigeration equipment has diverse structural forms and a wide range of applications.

[0025] The following description, in conjunction with the accompanying drawings, details the interaction method, interaction device, refrigeration equipment, and electronic equipment provided in this application through specific embodiments and application scenarios.

[0026] The interaction method of the refrigeration device provided in this application embodiment can be executed by an electronic device or a functional module or functional entity in an electronic device that can implement the interaction method of the refrigeration device. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras and wearable devices. The interaction method of the refrigeration device provided in this application embodiment will be described below using an electronic device as the execution subject.

[0027] like Figure 1 As shown, the interaction method of the refrigeration device includes steps 110, 120, 130, 140 and 150.

[0028] Step 110: Obtain interactive information.

[0029] Interactive information refers to the input data generated when a user interacts with the refrigeration equipment. Interactive information includes, but is not limited to, audio, text, or image information.

[0030] For example, the audio information in the interactive information could be a user asking the cooling device, "Can people with high blood pressure eat tomatoes?"

[0031] For example, the text information in the interactive information could be "query the expiration time of food" entered by the user on the touch screen of the refrigeration equipment.

[0032] For example, the image information in the interactive information could be a picture of the food that the user puts into the refrigeration device.

[0033] In practice, interactive information can be obtained by setting up corresponding acquisition devices on the cooling equipment. For example, a sound pickup device can be set up to obtain audio information, an image acquisition device can be set up to obtain image information, and a touch screen can be set up to obtain text information.

[0034] Step 120: When the interactive information includes audio information, extract the prosodic features of the interactive information.

[0035] Among them, prosodic features refer to the acoustic characteristics of a user's speech audio related to pitch, duration, stress, and intonation. Prosodic features can characterize the rhythm, intonation, and emotional expression of speech. Through prosodic features, the content of audio information can be captured and identified more accurately.

[0036] In some embodiments, prosodic features include at least one of intonation features, stress features, rhythm features, scale features, pitch features, and boundary dimension features.

[0037] Among them, intonation features refer to the pitch variation patterns in audio information. Intonation features are used to characterize the tone of a user’s speech, such as questioning, stating or exclaiming, and can reflect the overall intonation of the sentence.

[0038] Accent features refer to the degree to which certain syllables or words are emphasized in audio information. Accent features can be manifested as an increase in pitch, duration, or volume when speaking. Accent features are used to characterize the key points that a user wants to emphasize when speaking.

[0039] Rhythmic features refer to the temporal distribution patterns of syllables or words in a whole audio segment. Rhythmic features are used to characterize a user's speaking speed and pause patterns.

[0040] Pitch characteristics refer to the overall pitch range of audio information, and the pitch varies between different speakers or in different contexts.

[0041] High-pitched features refer to the absolute or relative pitch (i.e., the range of change) in an audio signal. High-pitched features can reflect the emotional state and emphasis of a user's speech.

[0042] Boundary dimension features refer to the boundaries of phrases or sentences in audio information. Boundary dimension features can be manifested as pitch drop, prolongation, or pause.

[0043] In this embodiment, prosodic features are extracted from the user's input audio information from different directions, such as intonation features, stress features, rhythm features, syllable features, pitch features, and boundary dimension features. This allows for a more comprehensive capture of the acoustic characteristics of the user's speech, enabling the cooling device to more accurately understand the user's intent, thereby improving the accuracy and naturalness of the recognition and interaction process.

[0044] Step 130: Based on prosodic features, determine multiple contextual features of different dimensions of audio information.

[0045] Among them, contextual features refer to feature vectors extracted from the prosodic features of audio information that can reflect the characteristics of speech in different dimensions. Contextual features can capture the contextual information of speech in terms of intonation, stress, etc. from various dimensions.

[0046] In this step, the multiple contextual features extracted based on prosodic features of different dimensions yield more accurate results compared to extracting them directly based on the original audio information.

[0047] Step 140: Based on multiple contextual features and the weight parameters corresponding to the multiple contextual features, obtain the semantic features of the audio information.

[0048] Among them, the weight parameters corresponding to multiple context features are used to measure the importance and contribution of different context features to the final semantic features.

[0049] Semantic features are vector representations obtained by weighted fusion of multiple contextual features. Semantic features can comprehensively reflect the core semantic content of audio information and can be used as input to a language model.

[0050] In this step, the semantic features of the audio information are determined through multiple contextual features of different dimensions. The importance and contribution of the information contained in different contextual features to the semantic features are taken into account, i.e., the weight parameters, so that the determined semantic features are more accurate.

[0051] Step 150: Input the semantic features of the audio information into the pre-trained language model so that the language model outputs the interaction results corresponding to the interaction information.

[0052] Among them, a language model, also known as a large language model, is a model that can generate natural language text or understand the meaning of language text and perform corresponding tasks.

[0053] In this step, semantic features reflect the interactive requests in the audio information input by the user, such as questions or instructions sent to the cooling system. By inputting the semantic features of the audio information into the language model, the interactive results output by the language model can be obtained, such as answering the user's questions or executing corresponding instructions, thus completing the entire interactive process from input interactive information to output interactive results. The accuracy of interactive recognition is high, and the user experience is good.

[0054] In related technologies, user input, such as voice input, is recognized through network models. Semantic feature information can be extracted from the user input, enabling the refrigeration equipment to have certain interactive capabilities and achieve more functions. However, in actual use, these technologies are prone to problems such as low recognition accuracy and slow response speed, which affects the user experience.

[0055] In this embodiment, prosodic features can characterize the speech rhythm, intonation, and emotional expression in audio information. Compared with related technologies that directly extract semantic features from user input, the interaction method provided in this embodiment first extracts prosodic features after obtaining the audio information input by the user, and then determines the context features based on the extracted prosodic features to obtain semantic features. This can more accurately capture and recognize the content of audio information, thereby improving the recognition accuracy and response speed in the interaction process and providing a better user experience.

[0056] According to the interaction method of the refrigeration device provided in the embodiments of this application, prosodic features can characterize the speech rhythm, intonation and emotional expression in audio information. By extracting prosodic features from audio information, the content of audio information can be captured and identified more accurately. The interaction results obtained based on prosodic features have high accuracy, fast response speed and good user experience.

[0057] In some embodiments, based on prosodic features, multiple contextual features of different dimensions of audio information are determined, including: Prosodic features are input into the first processing model to obtain multiple contextual features output by the first processing model. The first processing model includes multiple attention modules, which are used to obtain contextual features of prosodic features based on a bidirectional attention mechanism. The contextual features obtained by different attention modules have different dimensions.

[0058] The first processing model can be a feature extraction model based on a neural network. The first processing module is used to extract contextual features of multiple dimensions from prosodic features. The first processing model includes multiple attention modules, each of which is used to extract contextual features of different dimensions.

[0059] The attention module is the core structure of the first processing model. Each attention module can capture contextual information in prosodic features, suppress noise, and filter out important information or weights in each part of the prosodic features, thereby completing the extraction of contextual features.

[0060] Bidirectional attention is a method for extracting contextual features in attention modules. By using bidirectional attention, both preceding and following contextual information can be considered simultaneously, thereby obtaining more comprehensive contextual features.

[0061] In actual implementation, the first processing model can adopt the Transformer architecture. After the prosodic features are input into the first processing model, the query, key, and value matrices are obtained through a fully connected layer. Then, the query, key, and value matrices are divided into multiple attention heads through multiple attention modules. Each attention head has different dimensions and linear transformation parameters. Each attention module will perform independent bidirectional feature extraction on the corresponding attention head to obtain contextual features of different dimensions.

[0062] In this embodiment, the first processing model is constructed based on a bidirectional multi-head attention mechanism. The first processing model includes multiple attention modules. After the prosodic features are input into the first processing model, the multiple attention modules can run multiple independent attention mechanisms in parallel to analyze and extract the prosodic features. Each attention module runs a bidirectional attention mechanism, which can extract the prosodic features bidirectionally. Through the first processing model constructed based on the bidirectional multi-head attention mechanism, the contextual information included in the prosodic features can be extracted in multiple dimensions, and the obtained contextual feature results are accurate.

[0063] In some embodiments, the weight parameters corresponding to multiple context features are obtained through the following steps: Multiple context features are input into a second processing model to obtain weight parameters corresponding to the multiple context features output by the second processing model. The second processing model includes at least a first gated loop unit and a second gated loop unit. The first gated loop unit and the second gated loop unit are used to process multiple context features based on time order. The processing order of the first gated loop unit and the second gated loop unit is opposite.

[0064] The second processing model can be a weight calculation model based on a neural network. The second processing model is used to calculate the weight parameters corresponding to context features of different dimensions. The second processing model is constructed based on a bidirectional gated recurrent unit and contains two gated recurrent units with opposite processing orders.

[0065] Gated recurrent units (GRUs) are an improved recurrent neural network structure that controls the flow of information through update and reset gates. The update gate determines whether the current input updates the current state, while the reset gate determines how to combine the past state with the current input. Through update and reset gates, GRUs can effectively capture long-term dependencies in sequence data.

[0066] In this embodiment, the first gated loop unit and the second gated loop unit in the second processing model can process multiple contextual features based on the time series. The processing order of the first gated loop unit and the second gated loop unit is opposite. That is, the second processing model can perform bidirectional processing on multiple contextual features through two gated loop units, thereby capturing the contextual information of the sequence at the same time, providing a more comprehensive and accurate feature representation, and thus obtaining more accurate weight parameters.

[0067] Furthermore, the second processing model built based on the bidirectional gated loop unit has fewer parameters and consumes less computational resources, which can improve the response speed of the recognition and interaction process.

[0068] In some embodiments, the interactive information further includes text information, and the method further includes: Text enhancement processing is performed on the text information to obtain a first processing result. The text enhancement processing includes at least one of noise reduction and sparsity reduction. Feature extraction is performed on the first processing result to obtain the semantic features of the text information; The semantic features of the text information are input into a pre-trained language model so that the language model outputs the interaction results corresponding to the interaction information.

[0069] Among them, text information refers to the text content that users input into the refrigeration equipment through input devices such as touch screens, keyboards or remote terminals. The text information contains semantic expressions or operation instructions, such as "Can people with high blood pressure eat tomatoes?" or "Turn off the lights".

[0070] Text enhancement is a preprocessing step that optimizes the raw text information input by the user. Text enhancement can improve text quality and strengthen key features in the text information, providing more standardized input for subsequent semantic analysis.

[0071] In this embodiment, text enhancement processing may include noise reduction and sparsity reduction. Noise reduction can clean and correct irrelevant characters, typos, redundant spaces and other interfering factors in the text, thereby improving the purity of the text.

[0072] Sparsity processing compresses the feature space of text information through methods such as stemming and synonym merging, which can solve the problem of overly scattered text features, thereby filtering out useless words and increasing the density of core words in the text information.

[0073] In this embodiment, by extracting features from the text information in the interactive information, the diversity of the refrigeration preparation interaction method can be improved, enabling refrigeration preparation to be applied to more interactive scenarios. Furthermore, before extracting features from the text information, text enhancement processing is performed on the text information to optimize the text information, strengthen the key features in the text information, and make the extracted semantic features more accurate.

[0074] In some embodiments, the interaction information further includes image information, and the method further includes: The image information is encoded to obtain the second processing result; Based on the second processing result, determine the vectorized representation of the second processing result; Based on embedded representation, determine the semantic features of image information; The semantic features of the image information are input into a pre-trained language model so that the language model outputs the interaction results corresponding to the interaction information.

[0075] Image information refers to picture or video data captured by the refrigeration equipment through an image acquisition device. Image information may include food items that the user puts in or takes out of the refrigeration equipment.

[0076] The second processing result refers to the result obtained after binary encoding the original image information. Through binary encoding, key visual information or features in the image information can be transformed into a form that the program can understand, which is the second processing result.

[0077] Vectorized representation is the result obtained by vectorizing the second processing result. Through methods such as global pooling or feature aggregation, the high-dimensional features of the image (i.e., the second processing result) can be compressed into a low-dimensional vector representation (i.e., vectorized representation), which is convenient for subsequent semantic analysis and language model processing.

[0078] Variable embedding of special words refers to the technique of dynamically encoding specific categories of objects or key regions (such as food types, packaging labels, etc.) in image information during the process of vectorization. The variable embedding vector will adaptively adjust according to the image content, generating differentiated vector representations for important visual elements with different semantics, thereby capturing key information in the image more accurately.

[0079] In this embodiment, by extracting features from the image information in the interactive information, the diversity of the refrigeration preparation interaction method can be improved, enabling refrigeration preparation to be applied to more interactive scenarios. Furthermore, before extracting features from the image information, the image information is first encoded and vectorized, which can generate differentiated vector representations for important visual elements with different semantics in the image information, thereby capturing key information in the image more accurately and making the extracted semantic features more accurate.

[0080] In some embodiments, the language model includes a first sub-model and a second sub-model, wherein the response speed of the first sub-model is greater than that of the second sub-model, and the reasoning ability of the second sub-model is stronger than that of the first sub-model.

[0081] The first sub-model is a lightweight, fast-response model, which typically employs a simplified network structure or quantization techniques, sacrificing some accuracy in exchange for faster processing speed. It is suitable for scenarios with high real-time requirements.

[0082] The second sub-model is a high-performance deep model with a more complex network architecture and stronger feature extraction capabilities, enabling more accurate semantic understanding and reasoning tasks.

[0083] In practice, the first sub-model is suitable for existing application scenarios. The first sub-model can adopt a self-developed model such as the BERT model, which can be adapted to scenarios such as food management, food health knowledge graphs, and knowledge question answering. Alternatively, it can adopt a self-controlled large model, such as using DeepSeek-32B as the base model for model fine-tuning, or using DeepSeek-761B to distill the LLama model, etc. This not only reduces costs but also achieves the current requirement of fast response time. It can also be combined with the RAG architecture. For example, when facing a query, the results are calculated or extracted through semantic similarity, and then the required results are returned to the refrigeration equipment.

[0084] The second sub-model is suitable for tasks requiring deep thinking, such as reinforcement learning, visual language models, and complex reasoning tasks, involving deep thinking, learning without imitation, or learning without exploration. The second sub-model can use reinforcement learning training strategies such as DeepSeek-V3 base or R1 or open-source VLM models for model fine-tuning, model distillation into efficient models such as LLama / Qwen, and can also be combined with RAG architecture. For example, when facing a query, the results are calculated or extracted through semantic similarity, and the required results are returned to the refrigeration device. In addition, it can also obtain timely knowledge updates and a rich knowledge base.

[0085] In this embodiment, the semantic features of the audio information are input into a pre-trained language model so that the language model outputs the interaction result corresponding to the interaction information, including: If the complexity of the audio information is less than a preset complexity threshold, the interaction result is inferred through the first sub-model; Alternatively, if the complexity of the audio information is greater than or equal to a preset complexity threshold, the interaction result can be inferred through a second sub-model.

[0086] The preset complexity threshold is used to characterize the category of tasks or instructions contained in the interactive information. If the complexity of the audio information is less than the preset complexity threshold, it indicates that the task or instruction in the audio information belongs to a relatively simple and adapted application scenario. If the complexity of the audio information is greater than or equal to the preset complexity threshold, it indicates that the task or instruction in the audio information belongs to a relatively complex task that involves deep thinking, has not been learned, has not been imitated, or has not been explored.

[0087] In this embodiment, the language model is divided into a first sub-model and a second sub-model, which are responsible for quick response to simple tasks and deep thinking for complex tasks, respectively. Through the first sub-model and the second sub-model, the accuracy of interaction and response speed can be improved.

[0088] In some embodiments, the language model further includes a semantic embedding and feature fusion module.

[0089] In this embodiment, the semantic embedding and feature fusion module is used to embed semantic features corresponding to text information, audio information and image information (embedding token), then unify the semantic space through the mapping layer, and then align the resulting embedded semantic space. The semantic vectors of the resulting aligned space are then decoded to obtain the interaction result. The semantic embedding and feature fusion module can perform efficient feature fusion on multimodal semantic features.

[0090] In some embodiments, the pre-training process of the language model includes algorithm optimization and inference optimization.

[0091] In this embodiment, algorithm optimization can lead to a hybrid modular language model.

[0092] In practical implementation, modular algorithmic approaches can be adopted, such as targeted improvements to the attention mechanism. Depending on the target task, the decoder can be constructed using a feedforward network (FFN) and residual direct connection to obtain a hybrid expert model. For example, one shared expert can be used, while the other 255 experts can be used to process each category of application scenario separately, thereby improving interaction efficiency. In addition, the language model can be improved by model compression, multi-token prediction (MTP), multi-modal position rotation encoding, context learning algorithms, and reinforcement learning. The embodiments in this application are not limited here.

[0093] In this embodiment, inference optimization can complete the construction of the inference model in the language model.

[0094] In practice, process-supervised learning can be used, followed by sampling training strategies, such as reinforcement learning with proximal policy optimization (PPO) or direct preference optimization (DPO). Combined with knowledge retrieval, the obtained synthetic data is used to train preference optimization, thereby obtaining a complex task inference model. The base of the inference model can be DeepSeek-R1, LLama, Google open-source models, etc. Then, the optimal model base is selected by combining the training methods and their comparison results.

[0095] In some embodiments, the interactive results include at least one of voice broadcasting, text display, and image display.

[0096] In this embodiment, voice broadcasting can be achieved by converting the interaction result into a voice signal and playing it through an audio output device, and speech synthesis technology can be used to generate more natural and fluent voice feedback.

[0097] Text display presents the results of interactions in text form on a touch screen or remote terminal, such as a mobile phone or tablet.

[0098] Image display presents interactive information visually in the form of pictures, charts, or videos.

[0099] In this embodiment, the interaction results include at least one of voice broadcasting, text display, and image display, which enables the cooling device to output the interaction results in multiple ways, thereby improving the user experience.

[0100] The following is a specific example.

[0101] In this embodiment, the refrigeration device can be Figure 4 The refrigerator shown illustrates the interaction method of the refrigeration equipment through... Figures 2-3 The process shown is implemented.

[0102] Step 1: Complete the data collection for the interaction between the refrigeration equipment (such as a refrigerator) and the user. This includes internal and external environmental information, human-computer interaction, multi-turn dialogue, intelligent agents, multi-agent systems, personal intelligent assistants, data transmission, data generation tools, data uploading, and other hardware and software tasks related to intelligent terminals. The focus is on audio, text, image, and video data, including valid voice data (with and without wake-up words). Data requiring annotation or labeling, as well as image and video data, will be categorized and processed. In short, the data covers the user's interaction with the refrigerator's intelligent agent, such as retrieval, storage, and voice interaction modes. Image data is based on the refrigerator's spatial location ID pattern or structural space. This includes real-time and offline acquisition of voice, text, and video data from various input methods, including but not limited to intelligent devices or home appliances, such as 5G / 6G, WiFi, apps, microphones, mobile phones, and Bluetooth. If the hardware collector is an independent peripheral, the data collection will be completed by the external device.

[0103] Step 2: Complete the distillation and aggregation of various data, such as refrigeration equipment like refrigerators, to obtain results in text data form. This mainly targets the distillation and aggregation tasks for audio, images, and text.

[0104] Step 2.1: Audio information is converted into text. Currently, in intelligent interaction, voice data has a high word error rate and slow interaction response. Based on this, measures are taken to address audio data such as... Figure 2 As shown in Figure 102-2, a bidirectional gate controllable attention model method for prosodic speech enhancement is adopted: after acquiring effective speech, key features of prosody are extracted from the dimensions of intonation, stress, rhythm, pitch, and boundary; then, a multi-head bidirectional attention mechanism is input to obtain speech features focused on context and features in each dimension; then, a bidirectional gate controllable unit is input to obtain speech features with high context weight in each dimension and adjust and control them through the bidirectional gate; finally, the attention weights of each dimension feature are summed and calculated.

[0105] Step 2.2: Text information is processed through various methods, including text parsing, text prompting algorithms during prediction, and text enhancement generation data obtained through synthetic data or model- or data-driven methods. This is mainly because text data may contain noise or sparsity, which is not conducive to text feature extraction or calculation. The goal is to improve the performance of model algorithms and thus obtain more effective text data.

[0106] Step 2.3: Image information embedding semantic spaceization, that is, completing the encoding of image information and forming a unified embedding semantic space. First, the image or video is binary encoded; then, the image encoding is optimized for the length prompt of the expected target to make the image prompt more efficient; then, the variable embedding of special words is generated, that is, the embedding after image encoding, which can adopt the CLIP model, that is, image embedding tokenization.

[0107] Step 2.4, as follows Figure 2 As shown in 102-3, the corresponding data forms obtained from text, audio and image are first embedded (embedding tokens). Then, the semantic space is unified through the mapping layer. The resulting embedded semantic space is then aligned. The semantic vectors of the aligned space are then decoded to unify the dimension of the vector space and obtain efficient semantic feature fusion.

[0108] Step 3: Hybrid modular large language model, which involves completing the construction of the modules of the first and second sub-models with text data as the core.

[0109] Step 3.1: Complete the decoding task of the first sub-model. For existing application scenarios, use self-developed models such as the BERT model, which can be adapted to scenarios such as food management, food health knowledge graphs, and knowledge question answering. Alternatively, you can use self-controlled large models, such as using DeepSeek-32B as the base model for model fine-tuning, or using DeepSeek-761B to distill the LLama model, etc. This not only reduces costs but also increases response time. It can also be combined with the RAG architecture. For example, when facing a query, the results are calculated or extracted through semantic similarity, and then the required results are returned to the refrigerator.

[0110] Step 3.2: Complete the decoding task of the second sub-model. For tasks that require deep thinking, such as reinforcement learning, visual language models and their complex reasoning tasks, reinforcement learning training strategies such as DeepSeek-V3 base or R1 or open source VLM models are used to fine-tune the model and distill it into an efficient model, such as LLama / Qwen. It can also be combined with RAG architecture. For example, when facing a query, the result is obtained by calculating semantic similarity or extracting the result, and then the required result is returned to the refrigerator.

[0111] By constructing the above system, application scenarios can be classified and processed according to factors such as the frequency or number of interactive accesses, thereby improving the accuracy and effectiveness of interaction.

[0112] Step 3.3: Complete the computing power management and resource scheduling tasks, namely, conduct language model inference training and model training (including pre-training and post-training) through computing power resource scheduling. In order to better be compatible with the domestic computing power module, it can be divided into domestic computing power and high-end computing power modules, which is conducive to the management and use of computing power resources.

[0113] Step 4: Optimize the language model, including non-inference algorithm optimization such as... Figure 2 In section 104-1, the inference optimization module is as follows: Figure 2 104-2 in the middle.

[0114] Step 4.1: Non-inference algorithm optimization, which involves adopting modular algorithm methods, such as targeted improvements to the attention mechanism. Depending on the target task, the decoder is configured with FFN and residual direct connection to obtain a hybrid expert model. For example, one shared expert is used, and the other 255 experts are used to process each category of application scenario separately, thereby improving interaction efficiency. In addition, the language model can be improved by model compression, multimodal multi-token prediction, multimodal position rotation encoding, context learning algorithms, and reinforcement learning.

[0115] Step 4.2, Inference Optimization, involves using process-supervised learning and then reinforcement learning of the sampling training strategy PPO / DPO. Combined with knowledge retrieval, the obtained synthetic data is used to train preference optimization, thereby obtaining a complex task inference model. The base of the inference model can be DeepSeek-R1 or LLama, Google open-source models, etc. Then, the optimal model base is selected by combining the training methods and their comparison results.

[0116] Step 5: Complete the model backend, such as... Figure 2 105 refers to completing downstream multi-tasks such as engineering deployment and application service release of the pre-trained language model. This mainly involves configuring, releasing, and implementing data protection strategies for large model interfaces, fine-tuning the SFT+ reinforcement learning UI / UX interface, including voice, image, text, and video content, model encapsulation, model plugins, model service activation, interface call instructions, and interface result verification, to achieve a visual, user-friendly, and quantifiable multi-task joint application service interface.

[0117] Step 6: Complete the text-to-speech file conversion task, such as... Figure 2 The 107 in the example uses a natural language instruction-guided method to guide a context-customized synthesis model. Specifically, if there is a previous synthesis of the same timbre, the learned speaker's timbre is used for synthesis, resulting in a realistic sound effect. If there is contextual information, the synthesis is performed based on the guided content combined with natural language forms, resulting in a rich variety of synthesis effects, a natural and smooth MOS value, and strong intelligibility.

[0118] Step 7: Result Presentation, including tasks such as voice broadcasting, text and image presentation, etc. Figure 2 The 108 in the list, including speech and text output processing, are as follows: Step 7.1: For video output data, the decoding task can be completed through the Attention-based Convolutional Block (Att conV Block). The Attention-based Convolutional Block can be composed of a transformer deep network model with full attention module convolution (conV), rectified linear unit (ReLU), and layer normalization (LayerNorm). Referencing the specific implementation process: first, the full attention mechanism completes the feature vector or expression of the features themselves and between features, and then performs convolution operation, ReLU calculation, and LayerNorm. In this way, semantic features of high-level, long-dependency distance relationship are obtained to realize the decoding task and generate corresponding video data.

[0119] Step 7.2: Output text data format.

[0120] The various interactive services and modes are shown in Table 1.

[0121] Table 1

[0122] The interaction method for a refrigeration device provided in this application can be executed by an interaction device of the refrigeration device. This application uses the example of an interaction device of the refrigeration device executing the interaction method to illustrate the interaction device of the refrigeration device provided in this application.

[0123] This application also provides an interactive device for a refrigeration equipment.

[0124] like Figure 5 As shown, the interactive device of the refrigeration equipment includes: an acquisition module 510, a first processing module 520, a second processing module 530, a third processing module 540, and a fourth processing module 550.

[0125] The acquisition module 510 is used to acquire interactive information.

[0126] The first processing module 520 is used to extract the prosodic features of the interactive information when the interactive information includes audio information.

[0127] The second processing module 530 is used to determine multiple contextual features of different dimensions of audio information based on prosodic features.

[0128] The third processing module 540 is used to obtain the semantic features of the audio information based on multiple context features and the weight parameters corresponding to the multiple context features.

[0129] The fourth processing module 550 is used to input the semantic features of the audio information into the pre-trained language model so that the language model outputs the interaction results corresponding to the interaction information.

[0130] According to the interactive device of the cooling equipment provided in the embodiments of this application, prosodic features can characterize the speech rhythm, intonation and emotional expression in audio information. By extracting prosodic features from audio information, the content of audio information can be captured and identified more accurately. The interactive results obtained based on prosodic features have high accuracy, fast response speed and good user experience.

[0131] In some embodiments, the second processing module 530 is used to determine multiple contextual features of different dimensions of audio information based on prosodic features, including: Prosodic features are input into the first processing model to obtain multiple contextual features output by the first processing model. The first processing model includes multiple attention modules, which are used to obtain contextual features of prosodic features based on a bidirectional attention mechanism. The contextual features obtained by different attention modules have different dimensions.

[0132] In some embodiments, the third processing module 540 is further configured to input multiple context features into the second processing model to obtain weight parameters corresponding to the multiple context features output by the second processing model. The second processing model includes at least a first gated loop unit and a second gated loop unit. The first gated loop unit and the second gated loop unit are configured to process the multiple context features based on time order, and the processing order of the first gated loop unit and the second gated loop unit is opposite.

[0133] In some embodiments, the interactive information also includes text information, and the fourth processing module 550 is further configured to: Text enhancement processing is performed on the text information to obtain a first processing result. The text enhancement processing includes at least one of noise reduction and sparsity reduction. Feature extraction is performed on the first processing result to obtain the semantic features of the text information; The semantic features of the text information are input into a pre-trained language model so that the language model outputs the interaction results corresponding to the interaction information.

[0134] In some embodiments, the interactive information further includes image information, and the fourth processing module 550 is further configured to: The image information is encoded to obtain the second processing result; Based on the second processing result, determine the vectorized representation of the second processing result; Based on vectorized representation, determine the semantic features of image information; The semantic features of the image information are input into a pre-trained language model so that the language model outputs the interaction results corresponding to the interaction information.

[0135] In some embodiments, prosodic features include at least one of intonation features, stress features, rhythm features, scale features, pitch features, and boundary dimension features.

[0136] In some embodiments, the language model includes a first sub-model and a second sub-model, wherein the response speed of the first sub-model is greater than that of the second sub-model, and the reasoning ability of the second sub-model is stronger than that of the first sub-model. The fourth processing module 550 is used to input the semantic features of the audio information into a pre-trained language model, so that the language model outputs the interaction results corresponding to the interaction information, including: If the complexity of the audio information is less than a preset complexity threshold, the interaction result is inferred through the first sub-model; Alternatively, if the complexity of the audio information is greater than or equal to a preset complexity threshold, the interaction result can be inferred through a second sub-model.

[0137] In some embodiments, the interactive results include at least one of voice broadcasting, text display, and image display.

[0138] The interactive device for the refrigeration equipment provided in this application embodiment can achieve... Figures 2 to 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0139] This application also provides a refrigeration device, which includes the interaction device of the refrigeration device as described above.

[0140] According to the refrigeration device provided in the embodiments of this application, prosodic features can characterize the speech rhythm, intonation and emotional expression in audio information. By extracting prosodic features from audio information, the content of audio information can be captured and identified more accurately. The interaction results obtained based on prosodic features have high accuracy, fast response speed and good user experience.

[0141] In practice, refrigeration equipment can provide, for example... Figure 4 The refrigerator shown.

[0142] In some embodiments, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 801, it implements the various processes of the above-described interaction method embodiment of the cooling device and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0143] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0144] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described interaction method embodiment of the refrigeration device and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0145] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0146] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the interaction method of the above-mentioned refrigeration device.

[0147] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0148] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described interaction method embodiment of the refrigeration device, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0149] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0150] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0152] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0153] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0154] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. An interaction method for a refrigeration device, characterized in that, include: Obtain interactive information; When the interaction information includes audio information, the prosodic features of the interaction information are extracted; Based on the prosodic features, multiple contextual features of the audio information in different dimensions are determined; Based on the multiple context features and the weight parameters corresponding to the multiple context features, the semantic features of the audio information are obtained; The semantic features of the audio information are input into a pre-trained language model so that the language model outputs the interaction result corresponding to the interaction information.

2. The interaction method of the refrigeration equipment according to claim 1, characterized in that, The determination of multiple contextual features of different dimensions of the audio information based on the prosodic features includes: The prosodic features are input into a first processing model to obtain the multiple contextual features output by the first processing model. The first processing model includes multiple attention modules, which are used to obtain the contextual features of the prosodic features based on a bidirectional attention mechanism. The contextual features obtained by different attention modules have different dimensions.

3. The interaction method of the refrigeration equipment according to claim 1, characterized in that, The weight parameters corresponding to the multiple context features are obtained through the following steps: The multiple context features are input into the second processing model to obtain the weight parameters corresponding to the multiple context features output by the second processing model. The second processing model includes at least a first gated loop unit and a second gated loop unit. The first gated loop unit and the second gated loop unit are used to process the multiple context features based on time order. The processing order of the first gated loop unit and the second gated loop unit is opposite.

4. The interaction method of the refrigeration equipment according to claim 1, characterized in that, The interactive information also includes text information, and the method further includes: The text information is subjected to text enhancement processing to obtain a first processing result, wherein the text enhancement processing includes at least one of noise processing and sparsity processing; The semantic features of the text information are obtained by extracting features from the first processing result. The semantic features of the text information are input into a pre-trained language model so that the language model outputs the interaction result corresponding to the interaction information.

5. The interaction method of the refrigeration equipment according to claim 1, characterized in that, The interactive information also includes image information, and the method further includes: The image information is encoded to obtain a second processing result; Based on the second processing result, determine the vectorized representation of the second processing result; Based on the vectorized representation, the semantic features of the image information are determined; The semantic features of the image information are input into a pre-trained language model so that the language model outputs the interaction result corresponding to the interaction information.

6. The interaction method of the refrigeration equipment according to any one of claims 1-5, characterized in that, The prosodic features include at least one of intonation features, stress features, rhythm features, scale features, pitch features, and boundary dimension features.

7. The interaction method of the refrigeration equipment according to any one of claims 1-5, characterized in that, The language model includes a first sub-model and a second sub-model. The response speed of the first sub-model is greater than that of the second sub-model, and the reasoning ability of the second sub-model is stronger than that of the first sub-model. The step of inputting the semantic features of the audio information into a pre-trained language model, so that the language model outputs the interaction result corresponding to the interaction information, includes: If the complexity of the audio information is less than a preset complexity threshold, the interaction result is inferred through the first sub-model; Alternatively, if the complexity of the audio information is greater than or equal to a preset complexity threshold, the interaction result can be inferred through the second sub-model.

8. The interaction method of the refrigeration equipment according to any one of claims 1-5, characterized in that, The interactive results include at least one of voice broadcasting, text display, and image display.

9. An interactive device for a refrigeration equipment, characterized in that, include: The acquisition module is used to acquire interactive information; The first processing module is used to extract the prosodic features of the interactive information when the interactive information includes audio information; The second processing module is used to determine multiple contextual features of the audio information in different dimensions based on the prosodic features; The third processing module is used to obtain the semantic features of the audio information based on the multiple context features and the weight parameters corresponding to the multiple context features; The fourth processing module is used to input the semantic features of the audio information into a pre-trained language model so that the language model outputs the interaction result corresponding to the interaction information.

10. A refrigeration device, characterized in that, include: The interactive device of the refrigeration equipment as described in claim 9.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the interaction method of the refrigeration device as described in any one of claims 1-9.

Images