Control method and device of refrigeration equipment and refrigeration equipment

Abstract

The invention relates to the technical field of intelligent household electrical appliance control, and provides a control method and device of refrigeration equipment and the refrigeration equipment, and the control method comprises the following steps: responding to a trigger signal, and collecting space scene data and multi-modal data of the refrigeration equipment, the multi-modal data at least comprising image video data and voice data; extracting visual information and voice information in the multi-modal data; performing multi-modal fusion on the visual information and the voice information, and combining with the space scene data to confirm user requirements; and controlling the corresponding function modules to work according to user requirements. According to the method, more abundant and more-dimensional user intention information can be obtained by fusing two interaction modes of image video data and voice data and combining space scene data, so that the refrigeration equipment can more accurately understand composite user requirements including a target object, a space position and an operation instruction, and the user experience is improved. Therefore, the accuracy of control instruction generation and the reliability of interaction are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent home appliance control technology, and in particular to control methods, devices, and refrigeration equipment. Background Technology

[0002] With the development of smart home appliance technology, smart refrigeration devices with human-computer interaction functions, such as smart refrigerators, have gradually become widespread. Currently, user interaction with smart refrigerators mainly relies on physical buttons, touchscreens, and graphical menu interfaces. Users need to navigate through hierarchical menus or memorize specific operation paths to find and trigger the desired functions, such as adjusting the temperature of a specific storage compartment, setting a quick-cooling mode, or managing food. For some infrequently used or deeply hidden functions, users often need to spend a lot of time learning or searching, resulting in a cumbersome and inefficient interaction process and a poor user experience.

[0003] To enhance the intuitiveness and convenience of interaction, some technologies have adopted single-modal interactive control schemes. For example, some smart refrigerators support direct voice command control of certain functions, such as switching to refrigerator mode; others are equipped with visual sensors that can automatically input food information by recognizing specific ingredients or barcodes. However, these single-modal (pure voice or pure vision) interaction schemes have inherent limitations. In complex home environments with noise, voice recognition may misidentify or miss certain functions due to user accents, speech speed, or background interference; while pure vision schemes may fail to recognize functions due to changes in lighting conditions, food obstruction, or viewing angle issues. More importantly, single-modal systems provide limited information dimensions, making it difficult to accurately understand complex user commands that include spatial orientation, target objects, and specific operational intentions. This can lead to malfunctions or unresponsiveness in the refrigeration equipment, requiring users to make multiple corrections or revert to manual menu operations, thus limiting the improvement in intelligence and user experience. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention proposes a control method for refrigeration equipment to address the issues of limited information dimensions provided by a single mode in existing control processes, making it difficult to accurately understand complex instructions from the user's specific operational intentions, resulting in low levels of intelligence and a poor user experience.

[0005] This invention proposes a control method for a refrigeration device, the refrigeration device comprising several functional modules, and the control method for the refrigeration device comprising the following steps: In response to a trigger signal, spatial scene data and multimodal data of the refrigeration device are collected, wherein the multimodal data includes at least image and video data and voice data; Extract visual and speech information from the multimodal data; The visual information and the voice information are fused in a multimodal manner, and combined with the spatial scene data to confirm user needs; Based on the user's requirements, control the operation of the corresponding functional modules.

[0006] According to the control method for a refrigeration device proposed in this invention, the step of extracting visual and speech information from the multimodal data includes: The visual information, including gesture features and item category features, is extracted from the image and video data. The voice information is extracted from the voice data, and the voice information includes the operation intention based on the voice text.

[0007] According to the control method for a refrigeration device proposed in this invention, wherein the refrigeration device is a refrigerator, the step of multimodal fusion of the visual information and the voice information, combined with the spatial scene data to confirm user needs, includes: Based on the gesture characteristics and the spatial scene data, the spatial area that the refrigerator needs to control is determined; Based on the operational intent and the characteristics of the item category, determine the functional requirements associated with the spatial area and / or the characteristics of the item category; The spatial region, the item category characteristics, and the functional requirements are combined to generate the structured user requirements.

[0008] According to the control method for a refrigeration device proposed in this invention, the functional module includes a quick-freezing module. When the user requirements include functional requirements related to quick-freezing or quick-cooling, the steps for controlling the corresponding functional module to operate include: Based on the spatial region, the quick-freezing processing module at the corresponding physical location is activated; Based on the characteristics of the item category, a preset knowledge graph is invoked to determine the corresponding target rapid cooling time; After the target rapid cooling time has elapsed, a notification message indicating the end of rapid cooling is generated and feedback is provided.

[0009] According to the control method for refrigeration equipment proposed in this invention, the functional module includes a food management module. When the user requirements include functional requirements related to food management, the steps for controlling the corresponding functional module to operate include: Based on the visual information, identify the storage and retrieval status of the items corresponding to the item category features; Based on the access status, update the inventory information of the corresponding items in the refrigerator's database.

[0010] According to the control method for a refrigeration device proposed in this invention, the functional module includes a freshness assessment module. When the user requirement includes a functional requirement related to freshness or shelf life query, the steps for controlling the corresponding functional module to operate include: Obtain the shelf life information of the items corresponding to the item category characteristics; Based on the corresponding item, the stated shelf life information, and the corresponding storage time, assess the freshness level of the corresponding item and output the result.

[0011] According to the control method for a refrigeration device proposed in this invention, the functional module includes a recipe recommendation module. When the user's requirements include recipe-related functional requirements, the steps for controlling the corresponding functional module to operate include: Retrieve the types and quantities of food currently recorded in the refrigerator's database; Based on the correspondence in the preset knowledge graph and user preference information, generate multiple candidate recipes that match the types and quantities of the ingredients; The multiple candidate recipes are prioritized and then output according to the user preference information.

[0012] The present invention also proposes a control device for a refrigeration equipment, comprising: The data acquisition module is used to acquire spatial scene data and multimodal data of the refrigeration equipment in response to a trigger signal. The multimodal data includes at least image and video data and voice data. The feature extraction module is used to extract visual and speech information from the multimodal data; The fusion confirmation module is used to perform multimodal fusion of the visual information and the voice information, and combine them with the spatial scene data to confirm the user's needs; The control execution module is used to control the operation of the corresponding functional modules in the refrigeration equipment according to the user's requirements.

[0013] The present invention also proposes a refrigeration device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a control method for the refrigeration device.

[0014] The present invention also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a control method for the refrigeration device.

[0015] The present invention also proposes a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer is able to execute the control method of the refrigeration device.

[0016] The control method for refrigeration equipment provided by this invention, by fusing image and video data and voice data—two interaction modalities—and combining them with spatial scene data, can obtain richer and more multi-dimensional user intent information. This effectively overcomes the problems of misidentification, missed identification, or intent comprehension deviation that can easily occur when single-modal interaction is incomplete in information. It enables the refrigeration equipment to more accurately understand complex user needs that include target objects, spatial locations, and operational instructions, thereby significantly improving the accuracy of control command generation and the reliability of interaction.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of the overall control method for the refrigeration equipment provided in the embodiments of the present invention.

[0020] Figure 2 This is a schematic diagram of the process for extracting multimodal data provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of the multimodal fusion process provided in an embodiment of the present invention.

[0022] Figure 4 This is a flowchart of the quick-freezing processing module provided in an embodiment of the present invention.

[0023] Figure 5 This is a flowchart of the food ingredient management module provided in an embodiment of the present invention.

[0024] Figure 6 This is a flowchart of the freshness assessment module provided in an embodiment of the present invention.

[0025] Figure 7 This is a flowchart of the recipe recommendation module provided in an embodiment of the present invention.

[0026] Figure 8 This is a schematic diagram of the structure of the control device for the refrigeration equipment provided in an embodiment of the present invention.

[0027] Figure 9 This is a schematic diagram of the structure of the electronic equipment in the refrigeration device provided in the embodiment of the present invention.

[0028] Figure label: 810. Data acquisition module; 820. Feature extraction module; 830. Fusion confirmation module; 840. Control execution module; 910. Processor; 920. Communication interface; 930. Memory; 940. Communication bus. Detailed Implementation

[0029] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. To make the objectives, technical solutions, and advantages of the invention clearer, the technical solutions of the invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the invention, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0030] This invention provides a control method for a refrigeration device, the refrigeration device including several functional modules, such as... Figure 1 As shown, the control method for refrigeration equipment includes the following steps: Step S110: In response to the trigger signal, acquire spatial scene data and multimodal data of the cooling equipment. The multimodal data includes at least image and video data and voice data.

[0031] Trigger signals are typically generated by user interactions with refrigeration devices (such as smart refrigerators), such as the refrigerator door being opened, the user uttering a specific voice wake-up phrase like "Hello refrigerator," or activation via a gesture sensor. Once triggered, the refrigeration device immediately initiates multi-source information acquisition.

[0032] It primarily uses built-in depth sensors (such as Time of Flight (TOF) modules), 3D cameras, or vision-based depth estimation algorithms to acquire 3D spatial structure information of the device's interior or surrounding environment (such as storage compartments and drawers). Simultaneously, it can also capture real-time images of the device's interior or surrounding environment using an RGB camera, obtaining image and video data. For example, it can capture visual information such as the state of objects held by the user, object appearance features, and user gestures. Correspondingly, it can also collect user voice data via a microphone.

[0033] Step S120: Extract visual and speech information from the multimodal data.

[0034] After the data is collected, the raw data is initially analyzed and features are extracted to transform unstructured sensor data into structured or semi-structured semantic information, in preparation for intent understanding.

[0035] For example, computer vision models can be used to process image / video data. Automatic speech recognition technology can convert speech data into text, and natural language understanding models can be used to parse the text.

[0036] Step S130: Multimodal fusion of visual and voice information, combined with spatial scene data, to confirm user needs.

[0037] After extracting visual and speech information, the visual and speech information extracted in step S120 is aligned and fused with the spatial scene data obtained in step S110 within a unified context (time and space). For example, when the voice says "Turn this a little colder," and the visual system recognizes that the user's finger is pointing to a specific drawer and that a beverage is detected inside the drawer, the spatial data provides the drawer's location ID and adjustable settings.

[0038] The fusion inference engine correlates this information, eliminates ambiguity, and confirms that the user's request is not to adjust the temperature of the entire refrigerator, but rather to cool down the specific drawer containing the beverage. The result of the fusion inference can then be transformed into precise, structured user request instructions.

[0039] Step S140: Control the operation of the corresponding functional modules according to user needs.

[0040] Finally, the understood user needs are mapped to the specific physical or logical operations of the device.

[0041] Based on user requests, the corresponding functional modules are scheduled.

[0042] For example: If the requirement is quick-freezing, the quick-freezing module is invoked, sending control signals to the compressor and damper of the designated drawer to start the quick-freezing program. If the requirement is to record food items, the food management module is invoked, associating their storage location and storage time. If the requirement is to determine how many days they can be stored, the freshness assessment module is invoked, querying the storage time of the corresponding item, calculating and providing voice feedback on the remaining shelf life. Simultaneously or after the execution of the control command, the user is confirmed with the operation result through voice announcements, screen prompts, etc., such as "The milk in the left drawer has been quick-frozen, and you will be reminded in 30 minutes."

[0043] The control method for refrigeration equipment provided by this invention, by fusing image and video data and voice data—two interaction modalities—and combining them with spatial scene data, can obtain richer and more multi-dimensional user intent information. This effectively overcomes the problems of misidentification, missed identification, or intent comprehension deviation that can easily occur when single-modal interaction is incomplete in information. It enables the refrigeration equipment to more accurately understand complex user needs that include target objects, spatial locations, and operational instructions, thereby significantly improving the accuracy of control command generation and the reliability of interaction.

[0044] In some embodiments, such as Figure 2 As shown, step S120: the step of extracting visual and speech information from multimodal data, includes: Step S1210: Extract visual information from the image and video data. The visual information includes gesture features and item category features.

[0045] Specifically, for image and video data, video sequences or images can be analyzed. This process first detects and locates the user's hand area, then identifies specific gesture shapes (such as pointing with the index finger, open palm, clenched fist, etc.). Combining spatial scene data provided by a depth sensor or a monocular vision-based depth estimation algorithm, the two-dimensional image coordinates of the gesture are mapped to a three-dimensional spatial coordinate system inside the cooling device, thereby calculating the specific three-dimensional spatial coordinates of the fingertip or the center of the gesture. This gesture feature is ultimately structured into a data format of pointing coordinates (x, y, z) and the ID of the corresponding device area.

[0046] Simultaneously, object detection and image classification models run concurrently, performing real-time analysis on the same or multiple frames of images. The models will select all identifiable items in the image, especially those held by the user or clearly focused on, and predict their category, such as apples, milk cartons, or beef. The models can also output the confidence score of the item and its approximate location in the image. This item category feature is structured into data such as item ID and item category. In some advanced embodiments, this step can further identify the item's state characteristics, such as unopened, opened, or textual information, such as expiration date labels on packaging.

[0047] Step S1220: Extract speech information from the speech data, including the operation intent based on the speech text.

[0048] Specifically, for voice data, preprocessing such as noise reduction, enhancement, and frame segmentation can be performed on the collected raw voice data. Then, a speech recognition model is used to convert the audio stream into corresponding text transcription in real time. For example, the user's speech about freezing the beer for half an hour can be accurately converted into a corresponding text string.

[0049] After the text is transcribed, the natural language understanding module immediately performs deep semantic analysis based on a rule engine, a semantic parsing model, or a fine-tuned large language model. This module not only performs basic lexical and syntactic analysis, but also extracts structured operational intents from the text.

[0050] The operational intent here can include the operation action, target object, operation parameters, and specific location.

[0051] The user's desired action is clearly defined by actions such as setting, querying, starting, and adjusting. The target object, such as beer or vegetables in the left drawer, defines the physical entity to which the action is performed. Operational parameters, such as time and temperature parameters, specify the concrete requirements or conditions for the action. The specific location, such as the top or middle shelf, corroborates the spatial orientation in the visual information.

[0052] Ultimately, voice information can be transformed into operational intentions that include operation actions, target objects, operation parameters, and specific locations.

[0053] In some embodiments, such as Figure 3 As shown, the refrigeration device is a smart refrigerator. Step S130: The step of multimodal fusion of visual and voice information, combined with spatial scene data to confirm user needs, includes: Step S1310: Determine the space area that the refrigerator needs to control based on gesture characteristics and spatial scene data.

[0054] In this step, gesture features from step S1210 are received, including gesture type, such as pointing, and the coordinates of the fingertip in the camera coordinate system, as well as spatial scene data from step S110. Using a pre-calibrated coordinate transformation matrix, the fingertip coordinates in the gesture features are transformed from the image coordinate system to the refrigerator's 3D world coordinate system, consistent with the spatial scene data. Next, a ray or a cone-shaped region is calculated along the fingertip pointing direction, and collision detection or nearest neighbor calculation is performed with the bounding boxes of all defined functional areas in the refrigerator's 3D model. The functional area that intersects with or is closest to the gesture ray is determined. For example, the calculated result might be the target spatial area of ​​the refrigerator compartment, the third glass partition, the second position from the left, or the target spatial area being the upper drawer of the freezer compartment. If no explicit gesture is detected, inference can be made based on location modifiers in the voice information combined with a default strategy.

[0055] Step S1320: Based on the operational intent and item category characteristics, determine the functional requirements associated with the spatial area and / or item category characteristics.

[0056] In this step, the operation intention from step S1220 and the item category characteristics from step S1210 are received.

[0057] First, associate the target in the operation intention with the characteristics of the item category or a spatial region. For example, in the voice command "keep it fresh," the target needs to be determined by combining the visual recognition of strawberries to confirm that the operation object is a strawberry, rather than the entire spatial region.

[0058] Then, the built-in knowledge graph or rule base is invoked for reasoning. The knowledge graph stores relationships such as item category-recommendation storage conditions, supported functions in the functional area, and user historical preferences. An example of the reasoning process is as follows: For example, if the user's intention is to preserve freshness, the item category is strawberries, and the target space is the refrigerator's zero-degree preservation drawer, a knowledge graph query reveals that the recommended storage conditions for strawberries are high humidity and around zero degrees Celsius. The refrigerator's zero-degree preservation drawer supports temperature and humidity control. Furthermore, historical user data shows a preference for certain fruit and vegetable preservation modes.

[0059] The output function requirement is to enable the high-humidity zero-degree preservation mode for strawberries in the zero-degree preservation drawer of the refrigerator. This step can parse the user's vague control commands into specific, parameterized function call instructions within the device, thereby clarifying the functional requirements associated with spatial areas and / or item category characteristics.

[0060] Step S1330: Combine spatial regions, item category characteristics, and functional requirements to generate structured user requirements.

[0061] The system receives the target spatial region from step S1310 and the functional requirements for binding with item categories from step S1320. These are then organized according to predefined fields such as the operation object, operation location, execution action, and action parameters to form a complete structured data object. This object explicitly defines the object of the operation, the spatial target, the specific behavior, and the necessary constraints.

[0062] Furthermore, before generating the final instruction, the system can intelligently complete and verify the initially formed requirements based on a built-in knowledge model or user habit database. For example, when the functional requirement is quick-freezing but the duration is not specified, the system will automatically query the knowledge base for recommended quick-freezing time parameters for beef, and complete the instruction accordingly. Simultaneously, a consistency check will be performed. For instance, if the system detects a user gesture pointing to the refrigerated area, but the voice command and recognized item strongly suggest freezing, it may trigger a confirmation interaction with the user, or automatically correct the target space area according to preset priority rules to ensure the rationality and safety of the generated instruction.

[0063] After the above processing is completed, a standardized structured user requirement is finally output. This requirement is a set of machine-readable instructions that defines all dimensions of the entire control intent. It can be directly parsed by the smart refrigerator and accurately distributed to the corresponding modules for execution. Standardized structured user requirements can be, for example, "Left drawer, fruits, fruit preservation setting", "Right drawer, red meat, meat preservation setting", "Drawer, beverages, quick cooling setting", etc.

[0064] In one example, the functional modules include a quick-freezing processing module, such as... Figure 4 As shown, when user requirements include functional requirements related to quick freezing or quick cooling, the steps for controlling the operation of the corresponding functional modules include: Step S410: Based on the spatial region, start the quick-freezing processing module at the corresponding physical location.

[0065] In this embodiment, structured user requirements are parsed. A start command is sent to the quick-freezing module corresponding to the spatial area. For example, if a user points to a bottle in the refrigerator compartment and says "quick-freeze this," the entire freezer compartment will not be activated; instead, only the localized high-powered refrigeration unit associated with the shelf or area where that bottle is located will be activated.

[0066] Step S420: Based on the characteristics of the item category, call the preset knowledge graph to determine the corresponding target rapid cooling time.

[0067] In this embodiment, the system parses the item category characteristics from the structured user requirements, such as beverages, meats, or berries. Then, it accesses a domain knowledge graph built into the device or in the cloud. This knowledge graph stores rich physical property data and preservation rules, enabling it to retrieve recommended optimal cooling curves, target temperatures, and suggested quick-freezing times based on item category. For example, for beer, the knowledge graph might suggest cooling from room temperature to 4°C within XX minutes. The system sets this recommended value as the target quick-freezing time and configures the operating parameters of the quick-freezing module accordingly, thereby achieving customized quick-freezing based on food characteristics—scientific and energy-efficient—rather than simple, fixed-duration refrigeration.

[0068] Step S430: After the target rapid cooling time has elapsed, generate a notification message indicating that rapid cooling has ended and provide feedback.

[0069] The quick-freezing module starts a countdown upon startup. When the target quick-cooling time determined in step S420 is reached, the module not only automatically stops the quick-freezing mode but also actively generates a task completion notification. The feedback method is multimodal, such as broadcasting prompts through the device's voice synthesis module; simultaneously, the notification information may also be pushed to the user's associated mobile device app or displayed on the refrigerator's internal / door display screen.

[0070] In some embodiments, the notification may also include further intelligent suggestions, such as the beer being chilled and recommended to be consumed within 2 hours for the best taste, which can also be derived by querying consumption recommendations for the item in a knowledge graph.

[0071] In one example, the functional modules include an ingredient management module, such as... Figure 5 As shown, when user requirements include functional requirements related to food ingredient management, the steps to control the operation of the corresponding functional modules include: Step S510: Based on visual information, identify the storage and retrieval status of items corresponding to the item category characteristics.

[0072] In this embodiment, the food management module receives continuous frame analysis results from the visual processing pipeline, including the category of the item, its position trajectory in the image, and the interaction between the user's hand and the internal space of the refrigerator. By analyzing the changes of this information in a short time series, a pre-trained behavior recognition model or set logical rules are used to make judgments. For example, if an item is detected to enter from the external space of the refrigerator and eventually remain inside the refrigerator, it is determined to be in a stored state. If an item is detected to be moved from a known location inside the refrigerator (such as a shelf) and moved to the external space of the refrigerator with the user's hand, it is determined to be in a retrieved state.

[0073] Step S520: Update the inventory information of the corresponding items in the refrigerator's database according to the storage status.

[0074] In this embodiment, if the item is determined to be stored, the quantity is increased by 1 in the corresponding record of the item category in the food database, and the storage timestamp, storage location area, and optional visual feature snapshot are recorded simultaneously. If this is the first time this category has been entered, a new record is created. If the item is determined to be retrieved, a reduction operation is performed. The system attempts to correlate the retrieved item with the existing record in the inventory that best matches the storage location and deduct the amount. Whenever a new item is stored, the general shelf life is obtained from the knowledge graph based on its category, or by identifying the date on its packaging, and a countdown begins. As the inventory decreases or time progresses, the system can calculate and issue warnings for food items that are about to expire.

[0075] In addition, the database maintains real-time quantities of various ingredients. When the stock of a certain type of ingredient falls below a preset threshold (e.g., milk quantity is 0), a shopping list reminder can be automatically generated.

[0076] In one example, the functional modules include a freshness assessment module, such as... Figure 6 As shown, when user requirements include functionalities related to freshness or shelf-life queries, the steps for controlling the operation of the corresponding functional modules include: Step S610: Obtain the shelf life information of the item corresponding to the item category characteristics.

[0077] In this embodiment, item category features are received from structured user requirements. The built-in food database is accessed to obtain the generally recommended shelf life of that type of food under typical household refrigeration conditions, based on the item category. Furthermore, if the visual information of the food includes packaging, a date recognition model can be invoked to attempt to extract the printed production date, best-before date, or expiration date from the image. If the user manually entered the purchase date or expiration date via voice or touchscreen when the food was added to the database, the user-specified date is used. If no specific date is available, the average storage time of the same category of food in the database can be used as an empirical reference. The freshness assessment module performs a confidence assessment on these sources to ultimately determine the baseline shelf life time point used for evaluation.

[0078] Step S620: Combine the corresponding item, shelf life information and corresponding storage time to evaluate the freshness level of the corresponding item and output it.

[0079] First, the storage timestamp of the target item is retrieved from the database of the food management module. Then, the storage time is calculated. Freshness assessment is not a simple matter of determining "whether it has expired," but rather a dynamic decay model is established. The time difference between the current time and (storage time + shelf life) is calculated to determine the "remaining storage days."

[0080] Depending on the type of item, the model may incorporate compensation factors. For example, for leafy vegetables, the system may refer to their visual characteristics (such as analyzing whether the leaves are wilted or yellowed through regularly taken images) or combine historical data from the temperature and humidity sensors in the storage compartment (such as whether there are excessive temperature fluctuations) to make a weighted correction to the theoretical freshness.

[0081] Based on the above calculations and assessments, the freshness assessment module generates a freshness conclusion and provides feedback to the user in a multimodal manner. Freshness is quantified into several clear levels, such as fresh (safe to eat, high quality), near expiration (recommended for consumption), and not fresh (not recommended for consumption), while also providing a specific numerical value for "estimated shelf life of X days".

[0082] Finally, the evaluation results will be automatically synchronized to the food management database. When a food item is evaluated as "near its expiration date" or its freshness is rapidly declining, the freshness evaluation module can trigger a proactive reminder, which will be displayed on the refrigerator screen or sent to a mobile phone.

[0083] In one example, the functional modules include a recipe recommendation module, such as... Figure 7 As shown, when user requirements include recipe-related functionalities, the steps to control the operation of the corresponding functional modules include: Step S710: Obtain the types and quantities of food currently recorded in the refrigerator's database.

[0084] The recipe recommendation module accesses a real-time database maintained by the ingredient management module to obtain a detailed inventory list. This list includes not only the category names of all ingredients, but also their precise quantities (such as "200 grams", "2 pieces", "3 pieces") and optional freshness status.

[0085] In addition, special attention can be paid to ingredients that are nearing their expiration date or need to be used first, and these can be considered as key factors in the recommendation algorithm to help reduce food waste.

[0086] Step S720: Generate multiple candidate recipes that match the types and quantities of ingredients based on the corresponding relationships in the preset knowledge graph and user preference information.

[0087] In this embodiment, the knowledge graph not only contains a massive number of recipes, but also stores the relationships between ingredients and recipes, ingredient pairing rules, cooking techniques, and flavor characteristics. The engine first performs precise matching to find recipes where the required ingredients are completely covered by the current inventory. Next, it performs flexible matching, utilizing the ingredient substitution relationships in the knowledge graph or allowing for the absence of certain non-critical ingredients to generate more candidate solutions.

[0088] Step S730: Prioritize and output multiple candidate recipes according to user preference information.

[0089] Finally, by combining user preference information and comprehensively considering multiple dimensions such as ingredient matching degree, user preference relevance, cooking complexity, and even nutritional balance, a final comprehensive recommendation index is generated for each candidate recipe, and the results are sorted and output according to priority.

[0090] This invention also provides a control device for a refrigeration equipment, such as... Figure 8 As shown, the control device of the refrigeration equipment includes: a data acquisition module 810, a feature extraction module 820, a fusion confirmation module 830, and a control execution module 840.

[0091] In this embodiment, the data acquisition module 810 is used to acquire spatial scene data and multimodal data of the refrigeration equipment in response to a trigger signal. The multimodal data includes at least image and video data and voice data. The data acquisition module 810 is responsible for synchronously acquiring multi-dimensional, heterogeneous raw data under specific trigger conditions. The data acquisition module 810 continuously listens for or waits for the trigger signal. This signal can originate from the door magnetic sensor of the refrigeration equipment, a dedicated voice wake-up word recognition unit, or a specific gesture trigger sensor. Multi-source data is acquired synchronously; once triggered, the data acquisition module 810 immediately activates multiple data acquisition units in parallel. These include, for example, a visual acquisition unit, an auxiliary sensing unit, a voice acquisition unit, and a spatial perception unit. The visual acquisition unit typically includes one or more RGB cameras deployed at key locations inside the refrigerator to acquire image or video data of the scene, capturing user gestures, operational actions, and the appearance of food. The voice acquisition unit typically consists of a microphone array to acquire voice data. While capturing user voice commands, array technology can be used for sound source localization and background noise reduction. The spatial sensing unit may include a TOF sensor or a binocular camera to acquire three-dimensional spatial scene data inside the refrigerator, generating depth maps or point clouds to obtain accurate spatial structure information. The auxiliary sensing unit may integrate temperature and humidity sensors to acquire real-time environmental parameters for each storage area.

[0092] The feature extraction module 820 is used to extract visual and speech information from multimodal data. The feature extraction module 820 is responsible for the initial parsing of the raw data, extracting structured semantic features. The visual information extraction submodule integrates or calls computer vision algorithm models to process the received image / video data. The speech information extraction submodule integrates automatic speech recognition and preliminary natural language understanding functions. Its workflow involves converting speech to text, parsing the text, extracting operation verbs, target objects, positional modifiers, and parameters, forming a preliminary framework of operational intent.

[0093] The fusion confirmation module 830 is used to perform multimodal fusion of visual and voice information, combined with spatial scene data, to confirm user needs. After inference, the fusion confirmation module 830 generates structured user requirements. This requirement is a machine-readable instruction template that explicitly includes fields such as target area, target item, requested action, and action parameters.

[0094] The control execution module 840 is used to control the operation of corresponding functional modules in the refrigeration equipment according to user needs. The control execution module 840 receives structured user requests, first parses them, and determines the specific functional module to be invoked based on the request action field, such as the quick-freezing module, ingredient management module, or recipe recommendation module. Furthermore, the control execution module 840 is responsible for monitoring the execution status of the scheduled functional modules. After the function execution is completed, it may also be responsible for organizing feedback information and providing feedback to the user through the equipment's voice or display, completing the interaction loop.

[0095] like Figure 9 As shown, the cooling device includes electronic equipment comprising a processor 910, a communication interface 920, a memory 930, and a communication bus 940. The processor 910, communication interface 920, and memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute the following methods: in response to a trigger signal, acquiring spatial scene data and multimodal data of the cooling device, wherein the multimodal data includes at least image / video data and voice data; extracting visual and voice information from the multimodal data; performing multimodal fusion of the visual and voice information, and combining it with the spatial scene data to confirm user needs; and controlling the corresponding functional modules to operate according to the user needs.

[0096] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] On the other hand, embodiments of the present invention disclose a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: in response to a trigger signal, acquiring spatial scene data and multimodal data of the cooling device, wherein the multimodal data includes at least image and video data and voice data; extracting visual information and voice information from the multimodal data; performing multimodal fusion of the visual information and the voice information, and combining it with the spatial scene data to confirm user needs; and controlling the corresponding functional modules to work according to the user needs.

[0098] In another aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program is implemented to perform the transmission methods provided in the above embodiments, including, for example,: in response to a trigger signal, acquiring spatial scene data and multimodal data of the cooling device, wherein the multimodal data includes at least image and video data and voice data; extracting visual information and voice information from the multimodal data; performing multimodal fusion of the visual information and the voice information, and combining it with the spatial scene data to confirm user needs; and controlling the corresponding functional modules to operate according to the user needs.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only for illustrating the present invention and not for limiting the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention and should be covered within the scope of the claims of the present invention.

Claims

1. A control method for a refrigeration device, characterized in that, The refrigeration equipment includes several functional modules, and the control method of the refrigeration equipment includes the following steps: In response to a trigger signal, spatial scene data and multimodal data of the refrigeration device are collected, wherein the multimodal data includes at least image and video data and voice data; Extract visual and speech information from the multimodal data; The visual information and the voice information are fused in a multimodal manner, and combined with the spatial scene data to confirm user needs; Based on the user's requirements, control the operation of the corresponding functional modules.

2. The control method for the refrigeration equipment according to claim 1, characterized in that, The step of extracting visual and speech information from the multimodal data includes: The visual information, including gesture features and item category features, is extracted from the image and video data. The voice information is extracted from the voice data, and the voice information includes the operation intention based on the voice text.

3. The control method for the refrigeration equipment according to claim 2, characterized in that, The refrigeration device is a refrigerator. The process of multimodal fusion of the visual information and the voice information, combined with the spatial scene data to confirm user needs, includes: Based on the gesture characteristics and the spatial scene data, the spatial area that the refrigerator needs to control is determined; Based on the operational intent and the characteristics of the item category, determine the functional requirements associated with the spatial area and / or the characteristics of the item category; The spatial region, the item category characteristics, and the functional requirements are combined to generate the structured user requirements.

4. The control method for the refrigeration equipment according to claim 3, characterized in that, The functional module includes a quick-freezing module. When the user requirements include functional requirements related to quick-freezing or quick-cooling, the steps for controlling the operation of the corresponding functional module include: Based on the spatial region, the quick-freezing processing module at the corresponding physical location is activated; Based on the characteristics of the item category, a preset knowledge graph is invoked to determine the corresponding target rapid cooling time; After the target rapid cooling time has elapsed, a notification message indicating the end of rapid cooling is generated and feedback is provided.

5. The control method for the refrigeration equipment according to claim 3, characterized in that, The functional module includes a food ingredient management module. When the user requirements include functional requirements related to food ingredient management, the steps for controlling the operation of the corresponding functional module include: Based on the visual information, identify the storage and retrieval status of the items corresponding to the item category features; Based on the access status, update the inventory information of the corresponding items in the refrigerator's database.

6. The control method for the refrigeration equipment according to claim 3, characterized in that, The functional module includes a freshness assessment module. When the user's request includes a function related to freshness or shelf life query, the steps for controlling the operation of the corresponding functional module include: Obtain the shelf life information of the items corresponding to the item category characteristics; Based on the corresponding item, the stated shelf life information, and the corresponding storage time, assess the freshness level of the corresponding item and output the result.

7. The control method for the refrigeration equipment according to claim 3, characterized in that, The functional module includes a recipe recommendation module. When the user's requirements include recipe-related functional requirements, the steps for controlling the operation of the corresponding functional module include: Retrieve the types and quantities of food currently recorded in the refrigerator's database; Based on the correspondence in the preset knowledge graph and user preference information, generate multiple candidate recipes that match the types and quantities of the ingredients; The multiple candidate recipes are prioritized and then output according to the user preference information.

8. A control device for a refrigeration equipment, characterized in that, include: The data acquisition module is used to acquire spatial scene data and multimodal data of the refrigeration equipment in response to a trigger signal. The multimodal data includes at least image and video data and voice data. The feature extraction module is used to extract visual and speech information from the multimodal data; The fusion confirmation module is used to perform multimodal fusion of the visual information and the voice information, and combine them with the spatial scene data to confirm the user's needs; The control execution module is used to control the operation of the corresponding functional modules in the refrigeration equipment according to the user's requirements.

9. A refrigeration device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the control method of the refrigeration device as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the control method of the refrigeration device as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, enable the computer to perform the control method of the refrigeration device as described in any one of claims 1 to 7.

Images