Refrigerator screen control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a refrigerator screen control method and device, electronic equipment and a storage medium. The method comprises the steps that fast dynamic information, static images and space information of a user are obtained; acquiring infrared environment information of the refrigerator; generating intention prediction information and emotion prediction information for the user based on the rapid dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information; and the refrigerator screen is controlled through the intention prediction information and / or the emotion prediction information, the situation that a user can only interact with the refrigerator screen based on traditional modes such as touch control or voice is avoided, and the interaction flexibility and self-adaptability are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of refrigerator screen control, and in particular to a refrigerator screen control method, a refrigerator screen control device, an electronic device and a computer-readable storage medium. Background Art

[0002] The booming development of smart homes has promoted the increasing functionality of smart refrigerators, from the initial fresh-keeping storage to food management, shopping recommendations and even entertainment interaction. However, the current level of intelligence of smart refrigerators still has room for improvement, mainly reflected in the single interaction mode and the lack of user intention recognition and scene judgment. Existing products usually rely only on touch and voice commands for interaction, lacking flexibility and adaptability. More importantly, it is difficult for existing systems to accurately capture user intentions and judge usage scenarios, resulting in poor user experience. For example, when a user just passes by the refrigerator without intending to operate it, the screen may still remain on for a long time, causing energy waste; conversely, when the user needs to use the refrigerator function, the screen may fail to respond in time. The reason is that most existing smart refrigerators rely on a single type of sensor and use intention judgment logic based on fixed rules. Summary of the invention

[0003] The embodiments of the present invention provide a refrigerator screen control method, device, electronic device and computer-readable storage medium to overcome the above problems or at least partially solve the above problems.

[0004] The embodiment of the present invention discloses a refrigerator screen control method, comprising:

[0005] Obtain users' fast dynamic information, static images and spatial information;

[0006] Get infrared environment information of refrigerator;

[0007] Generate intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information;

[0008] The refrigerator screen is controlled by the intention prediction information and / or the emotion prediction information.

[0009] Optionally, the step of generating intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information includes:

[0010] Generating motion direction information for expressing the user's hand motion direction and eye motion direction based on the fast dynamic information;

[0011] determining distance information between the user and the refrigerator based on the spatial information;

[0012] Determining temperature change information of the user based on the infrared environment information;

[0013] Determining intention prediction information for the user through the motion direction information, and / or the distance information, and / or the temperature change information;

[0014] Emotion prediction information for the user is generated based on the static image.

[0015] Optionally, the step of controlling the refrigerator screen by using the intention prediction information and / or the emotion prediction information includes:

[0016] Controlling the refrigerator screen to perform a display triggering operation through the intention prediction information and / or the emotion prediction information;

[0017] Controlling the refrigerator screen to perform a display format adjustment operation through the intention prediction information and / or the emotion prediction information;

[0018] The intention prediction information and / or the emotion prediction information are used to control the refrigerator screen to execute display information push.

[0019] Optionally, the step of controlling the refrigerator screen to perform a display triggering operation using the intention prediction information and / or the emotion prediction information includes:

[0020] Determining trigger priorities and timing windows for the motion direction information, the distance information, the temperature change information, and the emotion prediction information;

[0021] When the motion direction information, and / or the distance information, and / or the temperature change information, and / or the emotion prediction information are obtained, the refrigerator screen is controlled to perform a display trigger operation based on the trigger priority and the timing window.

[0022] Optionally, the step of controlling the refrigerator screen to perform a display format adjustment operation using the intention prediction information and / or the emotion prediction information includes:

[0023] The font of the refrigerator screen is controlled to perform a zoom operation based on the motion direction information and the distance information.

[0024] Optionally, the step of controlling the refrigerator screen to perform a display format adjustment operation using the intention prediction information and / or the emotion prediction information includes:

[0025] The display format of the refrigerator screen adjustment interface is controlled based on the motion direction information and the distance information.

[0026] Optionally, the step of controlling the refrigerator screen to push display information using the intention prediction information and / or the emotion prediction information includes:

[0027] Determining the type of food currently displayed on the refrigerator screen;

[0028] When it is determined through the motion direction information that the time the user has viewed the food type exceeds a preset time threshold, and when it is determined that the emotion prediction information is confusion, associated information associated with the food type is pushed to the user.

[0029] Optionally, the step of controlling the refrigerator screen to push display information using the intention prediction information and / or the emotion prediction information includes:

[0030] Determine current display information of the refrigerator screen; the current display information includes necessary information and non-essential information;

[0031] When it is determined that the emotion prediction information is anxiety, the refrigerator screen is controlled to adjust the interface display format to delete the non-essential information and display the necessary information.

[0032] Optionally, the refrigerator is equipped with a visual perception sensor for acquiring the fast dynamic information, and further includes:

[0033] When it is determined through the infrared environment information that the time period during which the user has not interacted with the refrigerator exceeds a preset time threshold, the visual perception sensor is controlled to enter a low power consumption mode.

[0034] The embodiment of the present invention further discloses a refrigerator screen control device, comprising:

[0035] The initial information acquisition module is used to obtain the user's fast dynamic information, static images and spatial information;

[0036] An infrared environment information acquisition module is used to acquire infrared environment information of the refrigerator;

[0037] An intention and emotion prediction module, configured to generate intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information;

[0038] A refrigerator screen control module is used to control the refrigerator screen through the intention prediction information and / or the emotion prediction information.

[0039] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0040] The memory is used to store computer programs;

[0041] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0042] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0043] The embodiments of the present invention include the following advantages:

[0044] The embodiments of the present invention can bring the following significant beneficial effects by acquiring the user's fast dynamic information, static image and spatial information; acquiring the infrared environment information of the refrigerator; generating intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information; and controlling the refrigerator screen through the intention prediction information and / or the emotion prediction information:

[0045] 1. Improve the flexibility and adaptability of interaction:

[0046] Multimodal information fusion: By fusing fast dynamic information (such as gestures and body postures), static images (such as facial expressions and objects), spatial information (such as the distance and direction between the user and the refrigerator), and infrared environmental information (such as whether the human body is close or not), the system can perceive the user behavior and the environment more comprehensively and accurately.

[0047] More natural interaction: No longer limited to touch and voice, the system can automatically adjust the interaction method according to user behavior and scenarios. For example, when the user approaches the refrigerator, the screen automatically lights up; when the user points to an ingredient, the screen displays relevant information about the ingredient; when the user is confused, the system proactively provides help.

[0048] 2. More accurately identify user intent and determine usage scenarios:

[0049] Multi-dimensional information analysis: Combining dynamic, static, spatial and environmental information, the system can more accurately determine the user's true intention. For example, if the user is just passing by the refrigerator, the infrared sensor will detect the human body approaching, but combined with dynamic information such as the user's walking direction and speed, the system can determine that the user has no intention to operate, thereby avoiding the screen being turned on for a long time.

[0050] Addition of emotion recognition: By analyzing static image information such as facial expressions, the system can identify the user's emotional state and adjust the refrigerator's behavior based on the emotional state. For example, when the user shows anxiety, the system can simplify the interface and provide clearer operation instructions.

[0051] 3. Significantly reduce energy waste:

[0052] Accurate screen control: By being able to more accurately judge user intentions, the system can more intelligently control the screen on and off, avoiding unnecessary energy waste. For example, after the user leaves the refrigerator, the system can turn off the screen or enter low-power mode in time.

[0053] 4. Greatly improve response speed:

[0054] Proactive service: The system can proactively provide services based on user behavior and scenarios without the user having to do anything extra. For example, when a user opens the refrigerator door, the system can immediately display a list of ingredients or a recommended recipe.

[0055] Timely information push: When users are confused or need help, the system can push relevant information in a timely manner to improve user experience.

[0056] 5. Overcoming the limitations of single sensors and fixed rules:

[0057] Multi-sensor collaboration: Through the collaborative work of multiple sensors, the system can obtain richer and more comprehensive information, thereby understanding user behavior more accurately.

[0058] Intelligent intention and emotion prediction: Using smarter algorithms and models, the system can predict intention and emotion based on multi-dimensional information, overcoming the limitations of fixed rules and improving the flexibility and adaptability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flowchart of a refrigerator screen control method provided in an embodiment of the present invention;

[0060] Figure 2 is a structural schematic diagram of a smart refrigerator screen adaptive system based on a dynamic vision sensor provided in an embodiment of the present invention;

[0061] Figure 3 is a schematic diagram of a sensor structure of a smart refrigerator provided in an embodiment of the present invention;

[0062] Figure 4 is a structural block diagram of a refrigerator screen control device provided in an embodiment of the present invention;

[0063] Figure 5is a hardware structure block diagram of an electronic device provided in an embodiment of the present invention;

[0064] Figure 6 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] DVS sensor: visual perception sensor for capturing changes;

[0067] DVS sensor, full name Dynamic Vision Sensor, is a bionic vision sensor, and its working principle is inspired by the human visual system, especially the retina's response to light.

[0068] DVS sensors are different from traditional cameras. Instead of capturing images at a fixed frame rate, DVS sensors only generate events when the brightness of a pixel changes. In other words, it only focuses on movement and changes in the image, not the entire static scene.

[0069] High temporal resolution: DVS sensors have extremely high response speeds and can capture rapidly changing scenes with microsecond accuracy.

[0070] Low power consumption: Since data is generated only when there is a change, DVS sensors consume less power than traditional cameras.

[0071] High Dynamic Range: DVS sensors are able to adapt to extreme brightness changes, working well from dimly lit rooms to strong sunlight.

[0072] Working principle of DVS sensor:

[0073] Each pixel in the DVS sensor is independent. When the brightness of a pixel changes by more than a certain threshold, an event is generated. This event contains the location of the pixel, the polarity of the change (brightening or darkening), and the timestamp of the occurrence. These event data are transmitted in real time to form an event stream.

[0074] Advantages of DVS sensors:

[0075] Strong real-time performance: Due to its event-driven nature, DVS sensors are very suitable for applications that require real-time response, such as robot vision, drone vision, etc.

[0076] Motion perception: DVS sensors are very sensitive to motion and can accurately capture the movement trajectory and speed of objects.

[0077] Excellent performance in low light conditions: Due to the high dynamic range, DVS sensors provide reliable image information also in low light conditions.

[0078] Small data volume: Compared with traditional cameras, the amount of data generated by DVS sensors is much smaller, which is conducive to real-time processing and transmission.

[0079] Applications of DVS Sensors:

[0080] Robot vision: used for robot navigation, obstacle avoidance, grasping, etc.

[0081] Drone vision: used for drone flight control, target tracking, etc.

[0082] Augmented Reality: Used to track eye movements in real time to achieve a more immersive AR experience.

[0083] Scientific research: used to study visual systems, neuroscience and other fields.

[0084] DVS sensor is a new type of visual sensor, which has advantages that traditional cameras do not have. It is particularly suitable for visual applications that require high speed, low power consumption and high dynamic range. With the continuous development of technology, DVS sensor will play an important role in more and more fields.

[0085] Reference Figure 1 , shows a flowchart of a refrigerator screen control method provided in an embodiment of the present invention, which may specifically include the following steps:

[0086] Step 101, obtaining the user's fast dynamic information, static image and spatial information;

[0087] Step 102, obtaining infrared environment information of the refrigerator;

[0088] Step 103, generating intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information;

[0089] Step 104: controlling the refrigerator screen through the intention prediction information and / or the emotion prediction information.

[0090] refer to Figure 2 and Figure 3 , Figure 2 is a structural schematic diagram of a smart refrigerator screen adaptive system based on a dynamic vision sensor provided in an embodiment of the present invention; Figure 3 is a schematic diagram of a sensor structure of a smart refrigerator provided in an embodiment of the present invention;

[0091] In a specific implementation, an embodiment of the present invention can be applied to a smart refrigerator, which can be configured with a dynamic vision sensor, a traditional camera, a depth sensor, and an infrared sensor.

[0092] In the embodiment of the present invention, multimodal user data can be acquired by using a dynamic visual sensor in collaboration with other sensor perception and data acquisition modules;

[0093] Purpose: To build a comprehensive and accurate user portrait and establish a comprehensive user behavior model to provide a data basis for subsequent intent recognition and sentiment analysis.

[0094] Fast dynamic information: Through sensors such as DVS, the user's fast gestures, eye movements and other dynamic behaviors are captured to reflect the user's immediate intentions.

[0095] Static images: Capture the user's facial expressions through traditional cameras to reflect the user's emotional state.

[0096] Spatial information: The depth sensor is used to obtain the distance between the user and the refrigerator, spatial position information, and 3D information, providing a reference for subsequent behavior analysis.

[0097] Temperature change information: may be obtained through thermal imaging sensors, reflecting the thermal interaction between the user's body and the refrigerator, and indirectly reflecting the user's behavior.

[0098] Data meaning:

[0099] Gestures and eye movements: reflect the user's immediate intentions, such as opening and closing the refrigerator door and looking for food.

[0100] Facial expressions: reflect the user's emotional state, such as joy, anger, confusion.

[0101] Spatial information: reflects the relative position of the user and the refrigerator, which helps to determine whether the user is approaching or interacting with the refrigerator.

[0102] Beneficial effects:

[0103] Multi-dimensional perception: Capture user behavior more comprehensively through multimodal data fusion.

[0104] Strong real-time performance: Rapid acquisition of dynamic information ensures the system's real-time response to user behavior.

[0105] Rich data: Various types of data provide rich information for subsequent analysis.

[0106] The embodiment of the present invention can also obtain infrared environment information of the refrigerator;

[0107] Purpose: To understand the interaction between users and refrigerators and provide contextual information for subsequent intention prediction.

[0108] Based on temperature change information: Infer the direction of the user's movement by analyzing the thermal interaction between the user's body and the refrigerator.

[0109] Data meaning, that is, infrared environment information can include:

[0110] Refrigerator internal temperature distribution information: can reflect the placement of food in the refrigerator, temperature changes, and whether the user has opened the refrigerator door, etc.

[0111] Temperature change information used to determine whether the user is interacting with the refrigerator: By detecting the temperature changes around the user's body, it can be determined whether the user is approaching the refrigerator or touching the refrigerator door.

[0112] Beneficial effects:

[0113] Behavior analysis: provides important contextual information for subsequent intent prediction.

[0114] Interaction optimization: Knowing the user’s location enables more precise interactions. For example, when the user approaches a refrigerator, the screen brightness automatically increases.

[0115] The embodiment of the present invention can also generate intention prediction information and emotion prediction information through the emotion and intention prediction module;

[0116] Purpose: To understand the user’s needs and emotional state, and provide a basis for decision-making in subsequent interactions.

[0117] Intent prediction: Based on fast dynamic information, static images and spatial information, machine learning algorithms (such as RNN and CNN) are used to analyze user behavior and predict the user’s next action, for example:

[0118] The user stops in front of the refrigerator door, perhaps wanting to take some food.

[0119] The user slides his finger across the screen, perhaps wanting to adjust the temperature.

[0120] Emotion prediction: By analyzing the user's facial expressions and combining deep learning sentiment analysis algorithms, we can identify the user's emotional state, for example:

[0121] The user frowns, which may indicate confusion or dissatisfaction.

[0122] The user smiles, possibly indicating pleasure.

[0123] Beneficial effects:

[0124] Personalized service: Provide customized services based on user intentions and emotions.

[0125] Intelligent interaction: The system can proactively predict user needs and improve user experience.

[0126] Optionally, the refrigerator has an embedded intelligent computing system, including dynamic vision sensors, traditional cameras, depth sensors, and an embedded computing unit responsible for data processing. The tasks that complete the operation are highly real-time and have low computing requirements (such as gesture recognition and basic operation prediction) are completed by the refrigerator's embedded computing unit, while highly complex sentiment analysis and deep intention prediction can be uploaded to the cloud and returned by the server after calculation.

[0127] The embodiment of the present invention can also control the refrigerator screen according to the intention prediction information and / or the emotion prediction information as the prediction result;

[0128] Purpose: To dynamically adjust the refrigerator screen based on the prediction results to achieve intelligent interaction.

[0129] Intent-driven: Predict results based on intent and perform corresponding actions, for example:

[0130] When the user wants to get food, relevant recipes or ingredient information will be displayed.

[0131] The user wants to adjust the temperature, and the temperature adjustment interface is displayed.

[0132] Emotional response: Adjust the screen display according to the emotion prediction results, for example:

[0133] Provide more detailed instructions when users are confused.

[0134] When users are happy, recommend some interesting content.

[0135] Beneficial effects:

[0136] Improve user experience: Improve user satisfaction through personalized and intelligent interaction methods.

[0137] Enhanced interactivity: The refrigerator is no longer just a refrigeration device, but a smart home device with emotional interaction capabilities.

[0138] The embodiments of the present invention can bring the following significant beneficial effects by acquiring the user's fast dynamic information, static image and spatial information; acquiring the infrared environment information of the refrigerator; generating intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information; and controlling the refrigerator screen through the intention prediction information and / or the emotion prediction information:

[0139] 1. Improve the flexibility and adaptability of interaction:

[0140] Multimodal information fusion: By fusing fast dynamic information (such as gestures and body postures), static images (such as facial expressions and objects), spatial information (such as the distance and direction between the user and the refrigerator), and infrared environmental information (such as whether the human body is close or not), the system can perceive the user behavior and the environment more comprehensively and accurately.

[0141] More natural interaction: No longer limited to touch and voice, the system can automatically adjust the interaction method according to user behavior and scenarios. For example, when the user approaches the refrigerator, the screen automatically lights up; when the user points to an ingredient, the screen displays relevant information about the ingredient; when the user is confused, the system proactively provides help.

[0142] 2. More accurately identify user intent and determine usage scenarios:

[0143] Multi-dimensional information analysis: Combining dynamic, static, spatial and environmental information, the system can more accurately determine the user's true intention. For example, if the user is just passing by the refrigerator, the infrared sensor will detect the human body approaching, but combined with dynamic information such as the user's walking direction and speed, the system can determine that the user has no intention to operate, thereby avoiding the screen being turned on for a long time.

[0144] Addition of emotion recognition: By analyzing static image information such as facial expressions, the system can identify the user's emotional state and adjust the refrigerator's behavior based on the emotional state. For example, when the user shows anxiety, the system can simplify the interface and provide clearer operation instructions.

[0145] 3. Significantly reduce energy waste:

[0146] Accurate screen control: By being able to more accurately judge user intentions, the system can more intelligently control the screen on and off, avoiding unnecessary energy waste. For example, after the user leaves the refrigerator, the system can turn off the screen or enter low-power mode in time.

[0147] 4. Greatly improve response speed:

[0148] Proactive service: The system can proactively provide services based on user behavior and scenarios without the user having to do anything extra. For example, when a user opens the refrigerator door, the system can immediately display a list of ingredients or a recommended recipe.

[0149] Timely information push: When users are confused or need help, the system can push relevant information in a timely manner to improve user experience.

[0150] 5. Overcoming the limitations of single sensors and fixed rules:

[0151] Multi-sensor collaboration: Through the collaborative work of multiple sensors, the system can obtain richer and more comprehensive information, thereby understanding user behavior more accurately.

[0152] Intelligent intention and emotion prediction: Using smarter algorithms and models, the system can predict intention and emotion based on multi-dimensional information, overcoming the limitations of fixed rules and improving the flexibility and adaptability of the system.

[0153] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0154] In an optional embodiment of the present invention, the step of generating intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information includes:

[0155] Generating motion direction information for expressing the user's hand motion direction and eye motion direction based on the fast dynamic information;

[0156] determining distance information between the user and the refrigerator based on the spatial information;

[0157] Determining temperature change information of the user based on the infrared environment information;

[0158] Determining intention prediction information for the user through the motion direction information, and / or the distance information, and / or the temperature change information;

[0159] Emotion prediction information for the user is generated based on the static image.

[0160] The embodiment of the present invention can generate motion direction information based on fast dynamic information;

[0161] Purpose: Accurately capture the user's hand and eye movements to provide key clues for subsequent intention prediction.

[0162] Beneficial effects:

[0163] High accuracy: By directly measuring hand and eye movements, the user's intention can be captured more accurately. For example, if the user's hand moves quickly to the left while the eyes are looking at the upper left corner of the refrigerator, the system can infer that the user may want to take the item in the upper left corner.

[0164] Strong real-time performance: The collection of fast dynamic information can ensure the system's real-time response to user behavior.

[0165] The embodiment of the present invention can determine the distance information between the hand and the refrigerator based on the spatial information;

[0166] Purpose: To understand the interaction distance between the user and the refrigerator and provide a spatial reference for intent prediction.

[0167] Beneficial effects:

[0168] Distinguishing different operations: Through distance information, it is possible to distinguish whether the user is approaching the refrigerator or has already touched the refrigerator, thereby more accurately judging their intentions. For example, if the user's hand is very close to the refrigerator door and there is an action to open it, the system can infer that the user wants to open the refrigerator.

[0169] The embodiment of the present invention can determine the temperature distribution in the refrigerator and the temperature change of the hand based on the infrared environment information;

[0170] Purpose: To capture the heat distribution inside the refrigerator and the thermal interaction between the user and the refrigerator, and provide additional information for intent prediction.

[0171] Beneficial effects:

[0172] Fine-grained analysis: Temperature distribution information can help the system determine the area of ​​user attention. For example, if the user's hand stays in an area for a long time and the temperature in that area is low, the system can infer that the user may be looking for a cold drink.

[0173] Abnormality detection: By monitoring changes in hand temperature, some abnormal situations can be detected. For example, if the user's hand accidentally touches the freezer, the system can remind the user.

[0174] The embodiment of the present invention can generate intention prediction based on action direction, distance and temperature change information;

[0175] Purpose: To comprehensively analyze the collected multimodal data and infer the user's specific intentions.

[0176] Beneficial effects:

[0177] High accuracy: By combining data from multiple dimensions, the user's intention can be predicted more accurately. For example, if the user's hand moves to the right while the eyes are looking at the right compartment, and the hand temperature drops significantly, the system can determine with a high probability that the user wants to take a drink.

[0178] Strong flexibility: It can adapt to various complex user behaviors, such as users wandering in front of the refrigerator or opening multiple drawers at the same time.

[0179] Embodiments of the present invention can generate emotion predictions based on static images;

[0180] Purpose: To understand the user's emotional state by analyzing their facial expressions and provide support for personalized services.

[0181] Beneficial effects:

[0182] Improve user experience: Understanding the user's emotions can help the system respond more humanely. For example, if the user looks confused, the system can provide some suggestions to help the user find what they need.

[0183] Through comprehensive analysis of fast dynamic information, spatial information, and infrared environmental information, the system can accurately capture the user's movements, location, and interaction with the refrigerator. Combined with facial expression analysis, the system can more deeply understand the user's intentions and emotions, thereby providing more personalized and intelligent services. This multimodal fusion approach provides a new direction for the development of smart homes.

[0184] For example:

[0185] Scenario: The user returns home and opens the refrigerator door.

[0186] Data collection:

[0187] Fast Dynamic Message (DVS): The DVS sensor detects the user's hand moving toward the refrigerator door and then opening the refrigerator door.

[0188] Still image (camera): The camera captures the user's facial expression when he opens the refrigerator door. For example, if he smiles, it means he is in a good mood; if he frowns, it means he may be looking for a specific food.

[0189] Spatial information (depth sensor): The depth sensor measures the user’s distance from the refrigerator door and the position of his hand when he opens the refrigerator door.

[0190] Infrared environmental information (infrared sensor): The infrared sensor detects the temperature distribution inside the refrigerator and the temperature changes caused when the user's hand approaches the inner wall of the refrigerator.

[0191] Intent Prediction:

[0192] Based on DVS data: The system determines that the user's intention is to take food.

[0193] Based on static images: if the user is smiling, the system further infers that he may be looking for snacks; if he frowns, the system may infer that he is looking for specific ingredients.

[0194] Based on spatial information: The system determines the location where the user takes the food and recommends food or drinks in the adjacent area.

[0195] Based on infrared information: If the temperature in the refrigerator freezer is high, the system can remind the user to pay attention to the operating status of the refrigerator.

[0196] Sentiment prediction:

[0197] Based on static images: The system determines the user's emotional state based on his facial expressions. For example, if he smiles, the system determines that he is in a good mood; if he frowns, the system determines that he is in an upset mood.

[0198] Based on spatial information and infrared information: If the user frequently opens and closes the refrigerator door and has an anxious facial expression, the system can determine that he may be looking for a specific food and provide corresponding suggestions.

[0199] Finally, the system combines the above information to generate the following predictions:

[0200] Intent prediction: The user wants to find snacks.

[0201] Emotion prediction: The user is in a good mood.

[0202] System Response:

[0203] Refrigerator screen display: Recommend some snack options, such as potato chips, chocolate, etc.

[0204] Voice prompt: "Be careful when eating snacks at night. Try this new low-calorie yogurt."

[0205] In this example we see:

[0206] Multimodal data fusion: The system combines multiple data such as visual, spatial and thermal imaging to more accurately understand user intent.

[0207] Context-aware: The system not only pays attention to the user's current behavior, but also takes into account environmental information (such as the temperature inside the refrigerator) to provide more personalized services.

[0208] Emotional intelligence: The system can adjust the interaction mode according to the user's emotional state and provide more humane services.

[0209] Through comprehensive analysis of fast dynamic information, static images, spatial information and infrared environmental information, smart refrigerators can achieve more accurate predictions of user intentions and emotions, thereby providing more personalized and intelligent services.

[0210] In an optional embodiment of the present invention, the step of controlling the refrigerator screen by using the intention prediction information and / or the emotion prediction information includes:

[0211] Controlling the refrigerator screen to perform a display triggering operation through the intention prediction information and / or the emotion prediction information;

[0212] Controlling the refrigerator screen to perform a display format adjustment operation through the intention prediction information and / or the emotion prediction information;

[0213] The intention prediction information and / or the emotion prediction information are used to control the refrigerator screen to execute display information push.

[0214] The embodiment of the present invention can control the refrigerator screen to perform display triggering operations through intention prediction and emotion prediction;

[0215] Purpose: To proactively trigger the display on the refrigerator screen based on the user’s current intention and emotional state, thus achieving an intelligent interactive experience.

[0216] Beneficial effects:

[0217] Improve user experience: Users do not need manual operation, the screen can be automatically turned on according to their needs, improving convenience.

[0218] Save energy: Avoid leaving the screen in standby mode for long periods of time to reduce energy consumption.

[0219] Personalized service: By predicting user intentions, relevant information can be displayed in advance to meet user needs.

[0220] The embodiment of the present invention can control the refrigerator screen to perform display format adjustment operations through intention prediction and emotion prediction;

[0221] Purpose: To dynamically adjust the screen display format based on the different needs and emotions of users to provide a more personalized visual experience.

[0222] Beneficial effects:

[0223] Enhance user experience: Different display formats can adapt to different usage scenarios and user preferences, improving user satisfaction.

[0224] Improve the efficiency of information transmission: By adjusting the font size, color, layout, etc., information can be presented more clearly and its readability can be improved.

[0225] Implement multiple interaction methods: For example, different display methods can be used for different intentions, such as lists, cards, charts, etc.

[0226] The embodiment of the present invention can control the refrigerator screen to push display information through intention prediction and emotion prediction;

[0227] Purpose: Push personalized information to users based on their intentions and emotions to provide proactive services.

[0228] Beneficial effects:

[0229] Improve user stickiness: Increase the frequency of user interaction with the refrigerator by pushing valuable information.

[0230] Expand the scope of services: Not only limited to the information in the refrigerator, but also external information such as weather, news, etc. can be pushed.

[0231] Implement intelligent reminders: For example, when the milk in the refrigerator is about to expire, the system can proactively remind the user.

[0232] Through the above three steps, the refrigerator can achieve more intelligent and personalized interaction. For example:

[0233] Scenario 1: The user opens the refrigerator door, and the system determines that the user wants to find a cold drink through motion recognition and facial expression analysis. At this time, the screen automatically lights up and displays the cold drink category in large fonts and high contrast, while also pushing promotional information for recently purchased cold drinks.

[0234] Scenario 2: The user stands in front of the refrigerator, looking confused. The system determines that the user may be looking for a certain ingredient. The screen then displays all the ingredients in the refrigerator in a list and provides a smart search function.

[0235] Scenario 3: When the user returns home at night, the system automatically displays dinner suggestions for the day on the screen and provides corresponding recipes based on the user's previous habits.

[0236] In practical applications, realizing the above functions requires the comprehensive application of multiple technologies, including:

[0237] Computer vision: used for face recognition, action recognition, object recognition, etc.

[0238] Natural Language Processing: used to understand user intent and sentiment.

[0239] Machine learning: used to build predictive models to achieve intent prediction and sentiment prediction.

[0240] Human-computer interaction design: design user-friendly interaction interface and operation methods.

[0241] By making full use of the intention prediction and emotion prediction results, the refrigerator screen can achieve more intelligent and personalized interaction, providing users with a more convenient and comfortable experience. This intelligent interaction method not only improves the efficiency of refrigerator use, but also provides new ideas for the development of smart homes in the future.

[0242] In an optional embodiment of the present invention, the step of controlling the refrigerator screen to perform a display triggering operation through the intention prediction information and / or the emotion prediction information includes:

[0243] Determining trigger priorities and timing windows for the motion direction information, the distance information, the temperature change information, and the emotion prediction information;

[0244] When the motion direction information, and / or the distance information, and / or the temperature change information, and / or the emotion prediction information are obtained, the refrigerator screen is controlled to perform a display trigger operation based on the trigger priority and the timing window.

[0245] In practical applications, in order to ensure the reliability of screen startup, the embodiment of the present invention can design an efficient and user-friendly way to trigger the startup of the refrigerator screen based on the existing multimodal information (motion direction, distance, temperature change, emotion), and set a combination of multiple trigger conditions to adapt to different user behaviors.

[0246] Motion and distance based triggering:

[0247] Condition 1: The user's hand is close to the refrigerator door and there is an obvious opening action.

[0248] Condition 2: The user's hand stays near the refrigerator door for a long time and the distance is relatively close.

[0249] Trigger based on temperature change:

[0250] Condition 3: The temperature of the user's hand is significantly different from the temperature inside the refrigerator, and the hand is close to the refrigerator door.

[0251] Emotion and action-based triggers:

[0252] Condition 4: The user is smiling and stretching his hands towards the inside of the refrigerator.

[0253] Trigger logic:

[0254] Trigger priority: You can set priorities for different trigger conditions. For example, condition one has the highest priority and condition four has the lowest priority.

[0255] Timing Window: To avoid false triggering, you can set a timing window so that the screen starts only when multiple conditions are met simultaneously within a short period of time.

[0256] Specific implementation:

[0257] State machine: Create a state machine to manage the startup state of the screen. The initial state is standby. When the trigger condition is met, the state machine changes to active state and the screen lights up.

[0258] Threshold setting: For each trigger condition, set the corresponding threshold, for example, the distance threshold between the hand and the refrigerator, the temperature change threshold, etc.

[0259] Algorithm optimization: Machine learning methods can be used to train historical data to optimize trigger conditions and threshold settings.

[0260] Actions after the screen turns on:

[0261] Show welcome interface: Display a personalized welcome message, such as "Welcome back, XX".

[0262] Display commonly used functions: Display commonly used function buttons, such as viewing the ingredient list, adjusting the temperature, etc.

[0263] Display relevant content based on intent prediction: If the system has predicted the user's intention, it can directly display relevant content. For example, if the user wants to get a drink, the beverage category will be displayed directly.

[0264] For example:

[0265] Scenario 1: The user returns home and reaches for the refrigerator door. The system detects that the hand is close to the refrigerator door and is about to open it. The screen immediately lights up and displays the welcome interface and common functions.

[0266] Scenario 2: The user is busy in the kitchen and looks at the refrigerator from time to time. The system detects that her eyes are focused on the refrigerator door many times and the temperature of her hands is slightly increased. The screen automatically lights up and displays recipe recommendations.

[0267] In this way, a natural and intelligent way of starting the refrigerator screen can be achieved, greatly improving the user experience.

[0268] In the embodiment of the present invention, by determining the trigger priority and timing window for the action direction information, the distance information, the temperature change information and the emotion prediction information; when the action direction information, and / or the distance information, and / or the temperature change information, and / or the emotion prediction information are obtained, the refrigerator screen is controlled to perform a display trigger operation based on the trigger priority and the timing window, the following beneficial effects can be achieved.

[0269] User-friendly experience: There are various triggering methods, and users can choose different ways to start the screen according to their habits.

[0270] Low false trigger rate: The combination of multiple trigger conditions can effectively reduce the false trigger rate.

[0271] Strong adaptability: The trigger conditions and thresholds can be flexibly adjusted according to different users and scenarios.

[0272] In an optional embodiment of the present invention, the step of controlling the refrigerator screen to perform a display format adjustment operation through the intention prediction information and / or the emotion prediction information includes:

[0273] The font of the refrigerator screen is controlled to perform a zoom operation based on the motion direction information and the distance information.

[0274] In a specific implementation, the embodiment of the present invention can dynamically adjust the font size on the screen by analyzing the user's eye movement direction and the distance between the user and the refrigerator to adapt to the user's current interaction needs and visual habits.

[0275] For example,

[0276] Data collection and processing:

[0277] Movement direction information: The direction of the user's eye movement is obtained by analyzing the video stream or depth image.

[0278] Distance information: Use the depth sensor to measure the distance between the user and the screen.

[0279] Data preprocessing: Filter and smooth the collected data to remove noise and improve data accuracy.

[0280] Scaling strategy design:

[0281] Distance-based scaling: When the user is far away from the screen, the font size is increased so that the user can read clearly. When the user is close to the screen, the font size is reduced to avoid the font being too large and taking up too much screen space.

[0282] Scaling based on the direction of the action: If the user's hand moves closer to the screen, the font size can be further reduced to provide a more refined interactive experience.

[0283] If the user's hand is away from the screen and the distance is far, the font size can be further increased to facilitate user viewing.

[0284] Scaling that combines two factors: Taking into account the distance and direction of movement, the font size is adjusted dynamically. For example, when the user's hand approaches the screen and slides to the left, the font in the upper left corner can be enlarged to make it easier for the user to view the content in that area.

[0285] Scaling algorithm implementation:

[0286] Linear scaling: linearly adjusts the font size based on changes in distance or direction of action.

[0287] Non-linear scaling: To achieve a better user experience, a non-linear scaling function, such as an exponential function or a logarithmic function, can be used.

[0288] Adaptive scaling: Adaptively adjust the scaling ratio according to different scenarios and user preferences.

[0289] In other optional embodiments of the present invention, in addition to font scaling, this method can also be applied to the adjustment of other display elements, such as icon size, button position, etc., so as to achieve more flexible and diverse interface interactions.

[0290] By controlling font scaling based on motion direction and distance information, a more intelligent and personalized user interaction experience can be achieved. This approach can not only improve user reading comfort, but also provide more possibilities for future smart homes.

[0291] In an optional embodiment of the present invention, the step of controlling the refrigerator screen to perform a display format adjustment operation through the intention prediction information and / or the emotion prediction information includes:

[0292] The display format of the refrigerator screen adjustment interface is controlled based on the motion direction information and the distance information.

[0293] In an optional embodiment of the present invention, the step of controlling the refrigerator screen to execute display information push through the intention prediction information and / or the emotion prediction information includes:

[0294] Determining the type of food currently displayed on the refrigerator screen;

[0295] When it is determined through the motion direction information that the time the user has viewed the food type exceeds a preset time threshold, and when it is determined that the emotion prediction information is confusion, associated information associated with the food type is pushed to the user.

[0296] In an embodiment of the present invention, the screen adaptation and virtual interaction assistant module can monitor the time and emotional state of the user on the refrigerator screen. When the user shows a long-term attention to a certain type of food and has confused emotions, the system actively pushes relevant information, such as recipes, food combination suggestions, etc., to help the user solve the problem.

[0297] Exemplarily, this can be achieved in the following manner.

[0298] Food type identification:

[0299] Image recognition: Using computer vision technology, the food image currently displayed on the refrigerator screen is recognized and its category is determined.

[0300] Text matching: If the screen displays text information (such as the name of an ingredient), it is matched using natural language processing technology.

[0301] User behavior analysis:

[0302] Motion direction: By tracking the user's head or eye movements, it can be determined whether the user is looking at the current food type for a long time.

[0303] Dwell time: Calculates the time the user looks at the same type of food and compares it with the preset threshold.

[0304] Emotion recognition: Use facial expression recognition technology to determine whether the user shows emotions such as confusion, interest, etc.

[0305] Information push:

[0306] Acquisition of related information: Based on the identified food type, relevant recipes, ingredient combinations, cooking techniques and other information are obtained from the database.

[0307] Information filtering: Filter the acquired information and give priority to pushing information related to the user's historical behavior and preferences.

[0308] Push method: pop-up prompts, message notifications, etc.

[0309] Specific implementation example:

[0310] Scenario: A user browses vegetables on the refrigerator screen and stares at an eggplant for a long time with a frown.

[0311] System processing:

[0312] Identification: The system recognizes that the user is viewing eggplant.

[0313] Judgment: The system detects that the user has been staring at the eggplant for more than 10 seconds and his facial expression shows confusion.

[0314] Push: The system pushes a message: "Don't know how to cook eggplant? Try this eggplant box, simple and delicious!", with a recipe link attached.

[0315] By real-time monitoring of user behavior and emotional state, the system can proactively provide users with targeted information to help them solve problems and improve user experience. This information push method based on intention prediction and emotion prediction provides a new direction for the development of smart homes.

[0316] In an optional embodiment of the present invention, the step of controlling the refrigerator screen to execute display information push through the intention prediction information and / or the emotion prediction information includes:

[0317] Determine current display information of the refrigerator screen; the current display information includes necessary information and non-essential information;

[0318] When it is determined that the emotion prediction information is anxiety, the refrigerator screen is controlled to adjust the interface display format to delete the non-essential information and display the necessary information.

[0319] In a specific implementation, the embodiment of the present invention can simplify the display interface of the refrigerator screen, highlight necessary information, and hide unnecessary information when the system detects that the user is showing anxiety, so as to reduce the user's cognitive burden and provide a clearer experience. For example, if the user stays on a certain content interface for a long time and shows confusion (such as frowning, stagnant eyes), the system can not only pop up a window to recommend related recipes, ingredient combinations or operation prompts as in the previous example, but also simplify the interface at the same time, retaining only the core information related to the current content. For example, if the user is viewing the "Refrigerator Temperature Setting" interface, only the temperature adjustment button and the current temperature display are retained, and other irrelevant setting options are hidden, thereby helping the user to focus more on the current task.

[0320] Exemplarily, this can be achieved in the following manner.

[0321] Currently displayed information category:

[0322] Necessary information: refers to the most basic and core information required by the user for the current operation or task. For example:

[0323] Ingredients list: name, quantity, shelf life, etc. of food stored in the refrigerator.

[0324] Temperature setting: Temperature setting for each area of ​​the refrigerator.

[0325] Quick operation buttons: commonly used function entrances, such as "shopping list", "recipe recommendations", etc.

[0326] Non-essential information: refers to information that is not necessary for the current operation or task, such as advertising or promotional information;

[0327] Detailed recipe steps (only displayed if the user explicitly chooses to view the recipe);

[0328] Detailed parameters or settings of the refrigerator;

[0329] Decorative elements to beautify the interface.

[0330] Emotion Recognition:

[0331] Through facial expression recognition technology (such as analyzing the user's eyebrows, eyes, mouth and other features), the user's emotional state can be judged.

[0332] Optionally, other biosensor data (e.g., heart rate, skin conductance) can be combined to improve the accuracy of emotion recognition.

[0333] Interface adjustments:

[0334] Hide non-essential information: Remove or hide elements pre-defined as "non-essential information" from the screen.

[0335] Highlight necessary information: Enlarge the font, icon, or button of necessary information to make it more eye-catching.

[0336] Simplify layout: Use a simpler layout to reduce visual noise.

[0337] Color Adjustment: Use more muted colors and avoid using colors that are too bright or stimulating.

[0338] Specific implementation example:

[0339] Normal status: The refrigerator screen displays the ingredient list, recommended recipes, advertising information, etc.

[0340] Anxious state: The user quickly switches between different interfaces and his facial expressions show anxiety (for example, frowning frequently and looking around).

[0341] System processing:

[0342] Detection: The system detects that the user's emotional state is anxious.

[0343] Adjustment: The refrigerator screen automatically hides advertising information and infrequently used function buttons, and enlarges the font of the ingredient list to make it clearer and easier to read.

[0344] By dynamically adjusting the display information on the refrigerator screen based on emotion prediction, the cognitive burden of users in an anxious state can be effectively reduced, providing a more humane interactive experience. This approach not only improves the usability of the product, but also reflects the humanistic care of smart homes.

[0345] In an optional embodiment of the present invention, the refrigerator is equipped with a visual perception sensor for acquiring the fast dynamic information, and further includes:

[0346] When it is determined through the infrared environment information that the time period during which the user has not interacted with the refrigerator exceeds a preset time threshold, the visual perception sensor is controlled to enter a low power consumption mode.

[0347] In a specific implementation, the embodiment of the present invention can be based on edge computing and low-power control modules to judge the interaction status between the user and the refrigerator through infrared environmental information. When the user has not interacted with the refrigerator for a long time, the visual perception sensor is controlled to enter a low-power mode to reduce energy consumption and extend the service life of the equipment.

[0348] Exemplarily, this can be achieved in the following manner.

[0349] Infrared environmental information collection:

[0350] The refrigerator is equipped with an infrared sensor (such as an infrared pyroelectric sensor or an infrared imaging sensor) for detecting changes in thermal radiation in the environment surrounding the refrigerator.

[0351] When someone approaches the refrigerator, the infrared sensor detects the change in heat radiated by the human body; when the person leaves the refrigerator, the heat change disappears.

[0352] Determination of non-interaction duration:

[0353] The system records the timestamp of each time a user interaction is detected (such as opening a door, touching the screen, etc.).

[0354] The system periodically detects the time difference between the current time and the last interaction time, that is, the length of time without interaction.

[0355] The non-interaction time is compared with a preset time threshold (eg, 5 minutes, 10 minutes, etc.).

[0356] Sensor power mode control:

[0357] When the duration of no interaction exceeds the preset time threshold, the system determines that the user has not interacted with the refrigerator for a long time and controls the visual perception sensor to enter low power consumption mode.

[0358] Low power consumption modes can include the following:

[0359] Shutdown Sensor: Completely turns off the power to the visual perception sensor, stopping it from working.

[0360] Reduce the sampling rate: Reduce the frame rate at which the sensor collects images or videos to reduce the amount of data processing.

[0361] Sleep mode: puts the sensor into sleep mode and wakes up only when a specific signal is detected (for example, an infrared sensor detects a human body approaching).

[0362] Sensor wake-up:

[0363] When the infrared sensor detects a human body approaching the refrigerator again, the system determines that the user may want to interact with the refrigerator and immediately wakes up the visual perception sensor to restore it to normal working mode.

[0364] Specific implementation example:

[0365] Scenario: The user opens the refrigerator to take out food and leaves without going near the refrigerator again.

[0366] System processing:

[0367] Detection: When the infrared sensor cannot detect changes in human body thermal radiation, the system starts timing.

[0368] Judgment: After 5 minutes, the system determines that the duration of no interaction exceeds the preset threshold (assuming it is 5 minutes).

[0369] Control: The system controls the visual perception sensor to enter sleep mode.

[0370] Wake-up: Half an hour later, when the user approaches the refrigerator again, the infrared sensor detects the change in human body thermal radiation, and the system immediately wakes up the visual perception sensor.

[0371] In order to enable those skilled in the art to better understand the embodiment of the present invention, an example is used below to illustrate the embodiment of the present invention.

[0372] S11: Dynamic visual sensors collaborate with other sensors for perception and data collection. DVS captures the user's dynamic behavior (such as fast gestures, eye movements) in real time through an event-driven mechanism. Whenever the pixels of the user's hands, head or other parts change, DVS generates event data. After filtering, these event data are passed to the screen control system to respond to the user's dynamic operations in real time. Traditional cameras and depth sensors provide static monitoring, and the system comprehensively analyzes the user's status.

[0373] S12: Emotion and intention prediction. DVS captures the user's dynamic behavior (such as fast gestures and eye movements) in real time, traditional cameras and depth sensors provide static monitoring, and the system comprehensively analyzes the user's status. The fused sensor data is input into a deep learning model based on a neural network, which can predict the user's operation intention through training. By processing image data through a convolutional neural network (CNN), analyzing dynamic data of time series through a recurrent neural network (RNN), and combining with a reinforcement learning algorithm, the system can predict the next operation requirement based on the user's operation history and current behavior. For example, the system can identify whether the user wants to open a menu or switch the content on the refrigerator screen; the camera captures the user's facial expression, and combined with a deep learning emotion analysis algorithm, the system can identify the user's emotions (such as happiness, confusion, anxiety). Emotion analysis is used to adjust the display method of screen content. For example, when the system detects that the user is interested, it may extend the display time of a certain content.

[0374] Through the multimodal data fusion of dynamic vision sensors (DVS), traditional cameras and depth sensors, combined with deep learning models (CNN and RNN), accurate prediction of user intentions and emotions is achieved. Specifically, DVS uses an event-driven mechanism to capture users' dynamic behaviors (such as fast gestures and eye movements) in real time and generate event data; traditional cameras provide static images for facial expression analysis, and depth sensors provide spatial depth information to determine the distance and position between the user and the refrigerator screen. After time alignment and feature extraction, the fused multimodal data is input into the deep learning model for processing: CNN is used for image feature extraction, RNN analyzes the time series data of dynamic behaviors, and reinforcement learning algorithms are combined to predict users' operational intentions and emotional states.

[0375] Through sentiment analysis, the system can identify user emotions (such as pleasure, confusion, anxiety) and infer user needs based on operation history. For example, when the system detects that the user is confused, it can extend the display time of the current content or provide more detailed information prompts; when the user is happy, the system may recommend more interactive functions or content. In addition, based on intent prediction, the system can identify whether the user wants to switch interfaces or open a certain functional module, and adjust the screen content display in real time, such as enlarging the font when approaching the screen, or simplifying the interface when switching quickly. In this way, the system realizes screen adaptive control and intelligent interaction based on user emotions and behaviors, improving the accuracy and personalization of the user experience.

[0376] S13: Edge computing and low power control. The edge computing unit processes the DVS event stream in real time, combines the static sensor data, makes decisions quickly and feeds back to the screen display control system; when the user does not interact with the refrigerator for a long time, the DVS enters low power mode, the traditional camera and depth sensor reduce the working frequency, and the system automatically adjusts to save energy. The data generated by each sensor is fused in real time on the local edge computing unit. After combining the dynamic data provided by the DVS, the static image of the camera, and the spatial information of the depth sensor, the user's overall behavior and emotional state are analyzed through a deep learning model. The system uses a multimodal data fusion algorithm to synchronously process heterogeneous data from different sensors to generate a complete behavior model of the user. The model not only includes the user's dynamic behavior, but also includes static status and spatial position.

[0377] S14: Screen adaptation and virtual interaction. The screen adaptively adjusts the content display according to the user's dynamic behavior and static state, and the virtual assistant combines the user's emotions and operation intentions to provide personalized intelligent recommendations. Based on the output of the intention prediction model, the system adjusts the display content on the refrigerator screen in real time.

[0378] Ways to make intelligent recommendations based on user sentiment include:

[0379] The core of intelligent recommendation based on user emotions is to achieve dynamic content adjustment through the combination of emotion perception and personalized recommendation. The system captures multimodal data such as user facial expressions and dynamic behaviors through cameras and sensors, and analyzes emotional states (such as pleasure, doubt, and anxiety) in combination with deep learning models. The results of sentiment analysis serve as a dynamic adjustment factor for the recommendation algorithm to optimize content recommendations, such as recommending new content or interesting videos when the user is happy, providing intuitive information when confused, and simplifying the interface when anxious. The system continuously optimizes sentiment weights and recommendation strategies through reinforcement learning.

[0380] If the user is viewing multiple items quickly, the system will automatically adjust the screen layout to adapt it to the current operating environment. For example, when the user is close to the screen, the display font will automatically increase; when the user is far away from the screen, the system will simplify the interface display.

[0381] Personalized content recommendation: Based on the user's historical operations, current emotional state and environmental information, the system can dynamically recommend relevant content. For example, if the system detects that the user has stayed on a food information interface for a long time, it may automatically recommend related cooking plans or ingredient combinations. Through sentiment analysis, user historical behavior data and AI recommendation system, the system can achieve accurate personalized recommendations. This module combines reinforcement learning to continuously optimize the recommendation effect.

[0382] Through multimodal perception, deep learning prediction and intelligent interaction, the system fully realizes adaptive adjustment of the screen and personalized recommendations of the virtual assistant. The system combines dynamic vision sensors (DVS), traditional cameras and depth sensors to capture user behavior and emotional state in real time: DVS monitors dynamic behaviors such as fast gestures and eye movements, the depth sensor provides the spatial distance and position data between the user and the screen, and the camera analyzes the user's facial expressions to judge emotions. After time alignment and feature extraction, the fused multimodal data is input into the deep learning model (CNN extracts image features, RNN analyzes time series data), combined with the reinforcement learning algorithm to accurately predict the user's current operation intention and emotional state.

[0383] When the user approaches the screen, the system will automatically enlarge the font and adjust the screen layout to highlight the core information. If the user stays on a certain content interface for a long time and shows confusion (such as frowning or stagnant eyes), the virtual assistant will pop up a window to recommend related recipes, ingredient combinations or operation tips.

[0384] When the user switches content quickly and his facial expression shows anxiety, the system hides secondary information by simplifying the interface and retains only core functions (such as a list of ingredients in the refrigerator or quick operation buttons).

[0385] In addition, the system combines the user's historical operation data, real-time emotional state and environmental context (such as time, season) to dynamically generate personalized recommendation content, such as recommending suitable recipes or healthy diet plans based on the current time. When the user is not operating, DVS and other sensors enter low-power mode to save energy. All these functions are completed in real time by the edge computing unit to ensure low-latency response. At the same time, the user behavior model is optimized through multimodal data fusion and reinforcement learning, and the intelligence and personalization level of screen interaction are continuously improved, thereby significantly improving the user experience and efficiency.

[0386] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0387] Reference Figure 4 , shows a structural block diagram of a refrigerator screen control device provided in an embodiment of the present invention, which may specifically include the following modules:

[0388] The initial information acquisition module 401 is used to acquire the user's fast dynamic information, static image and spatial information;

[0389] Infrared environment information acquisition module 402, used to acquire infrared environment information of the refrigerator;

[0390] The intention and emotion prediction module 403 is used to generate intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information;

[0391] The refrigerator screen control module 404 is used to control the refrigerator screen through the intention prediction information and / or the emotion prediction information.

[0392] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0393] In addition, an embodiment of the present invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned refrigerator screen control method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0394] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the refrigerator screen control method embodiment described above is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0395] Figure 5 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0396] The electronic device 500 includes but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 510, and a power supply 511. Those skilled in the art will appreciate that Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0397] It should be understood that in the embodiment of the present invention, the radio frequency unit 501 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 510 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 501 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 501 can also communicate with the network and other devices through a wireless communication system.

[0398] The electronic device provides users with wireless broadband Internet access through the network module 502, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0399] The audio output unit 503 can convert the audio data received by the RF unit 501 or the network module 502 or stored in the memory 509 into an audio signal and output it as sound. Moreover, the audio output unit 503 can also provide audio output related to a specific function performed by the electronic device 500 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 503 includes a speaker, a buzzer, a receiver, etc.

[0400] The input unit 504 is used to receive audio or video signals. The input unit 504 may include a graphics processor (GPU) 5041 and a microphone 5042. The graphics processor 5041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 506. The image frame processed by the graphics processor 5041 can be stored in the memory 509 (or other storage medium) or sent via the radio frequency unit 501 or the network module 502. The microphone 5042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 501 in the case of a telephone call mode.

[0401] The electronic device 500 also includes at least one sensor 505, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 5061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 5061 and / or the backlight when the electronic device 500 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 505 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0402] The display unit 506 is used to display information input by the user or information provided to the user. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0403] The user input unit 507 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 507 includes a touch panel 5071 and other input devices 5072. The touch panel 5071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 5071 or near the touch panel 5071 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 5071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the contact point coordinates, and then sends it to the processor 510, receives the command sent by the processor 510 and executes it. In addition, the touch panel 5071 can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 5071, the user input unit 507 may also include other input devices 5072. Specifically, other input devices 5072 may include but are not limited to a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail here.

[0404] Furthermore, the touch panel 5071 may be overlaid on the display panel 5061. When the touch panel 5071 detects a touch operation on or near it, it transmits the information to the processor 510 to determine the type of the touch event. Then, the processor 510 provides a corresponding visual output on the display panel 5061 according to the type of the touch event. Figure 5 In the figure, the touch panel 5071 and the display panel 5061 are used as two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 5071 and the display panel 5061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0405] The interface unit 508 is an interface for connecting an external device to the electronic device 500. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 508 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 500 or may be used to transmit data between the electronic device 500 and an external device.

[0406] The memory 509 can be used to store software programs and various data. The memory 509 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 509 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0407] The processor 510 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 509 and calling data stored in the memory 509, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 510 may include one or more processing units; preferably, the processor 510 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 510.

[0408] The electronic device 500 may also include a power source 511 (such as a battery) for supplying power to various components. Preferably, the power source 511 may be logically connected to the processor 510 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0409] In addition, the electronic device 500 includes some functional modules not shown, which will not be described in detail here.

[0410] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0411] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0412] like Figure 6 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 601 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the refrigerator screen control method described in the above embodiment.

[0413] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0414] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0415] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0416] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0417] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0418] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0419] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0420] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A refrigerator screen control method, characterized in that: include: Obtain users' fast dynamic information, static images and spatial information; Get infrared environment information of refrigerator; Generate intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information; The refrigerator screen is controlled by the intention prediction information and / or the emotion prediction information.

2. The method according to claim 1, characterized in that The step of generating intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information comprises: Generating motion direction information for expressing the user's hand motion direction and eye motion direction based on the fast dynamic information; determining distance information between the user and the refrigerator based on the spatial information; Determining temperature change information of the user based on the infrared environment information; Determining intention prediction information for the user through the motion direction information, and / or the distance information, and / or the temperature change information; Emotion prediction information for the user is generated based on the static image.

3. The method according to claim 2, characterized in that The step of controlling the refrigerator screen by using the intention prediction information and / or the emotion prediction information includes: Controlling the refrigerator screen to perform a display triggering operation through the intention prediction information and / or the emotion prediction information; Controlling the refrigerator screen to perform a display format adjustment operation through the intention prediction information and / or the emotion prediction information; The intention prediction information and / or the emotion prediction information are used to control the refrigerator screen to execute display information push.

4. The method according to claim 3, characterized in that The step of controlling the refrigerator screen to perform a display triggering operation through the intention prediction information and / or the emotion prediction information includes: Determining trigger priorities and timing windows for the motion direction information, the distance information, the temperature change information, and the emotion prediction information; When the motion direction information, and / or the distance information, and / or the temperature change information, and / or the emotion prediction information are obtained, the refrigerator screen is controlled to perform a display trigger operation based on the trigger priority and the timing window.

5. The method according to claim 3, characterized in that: The step of controlling the refrigerator screen to perform a display format adjustment operation through the intention prediction information and / or the emotion prediction information includes: The font of the refrigerator screen is controlled to perform a zoom operation based on the motion direction information and the distance information.

6. The method according to claim 3, characterized in that The step of controlling the refrigerator screen to perform a display format adjustment operation through the intention prediction information and / or the emotion prediction information includes: The display format of the refrigerator screen adjustment interface is controlled based on the motion direction information and the distance information.

7. The method according to claim 3, characterized in that The step of controlling the refrigerator screen to push display information through the intention prediction information and / or the emotion prediction information includes: Determining the type of food currently displayed on the refrigerator screen; When it is determined through the motion direction information that the time the user has viewed the food type exceeds a preset time threshold, and when it is determined that the emotion prediction information is confusion, associated information associated with the food type is pushed to the user.

8. The method according to claim 3, characterized in that The step of controlling the refrigerator screen to push display information through the intention prediction information and / or the emotion prediction information includes: Determine current display information of the refrigerator screen; the current display information includes necessary information and non-essential information; When it is determined that the emotion prediction information is anxiety, the refrigerator screen is controlled to adjust the interface display format to delete the non-essential information and display the necessary information.

9. The method according to claim 1, characterized in that: The refrigerator is equipped with a visual perception sensor for acquiring the fast dynamic information, and further includes: When it is determined through the infrared environment information that the time period during which the user has not interacted with the refrigerator exceeds a preset time threshold, the visual perception sensor is controlled to enter a low power consumption mode.

10. A refrigerator screen control device, characterized in that: include: The initial information acquisition module is used to obtain the user's fast dynamic information, static images and spatial information; An infrared environment information acquisition module is used to acquire infrared environment information of the refrigerator; An intention and emotion prediction module, configured to generate intention prediction information and emotion prediction information for the user based on the fast dynamic information, and / or the static image, and / or the spatial information, and / or the infrared environment information; A refrigerator screen control module is used to control the refrigerator screen through the intention prediction information and / or the emotion prediction information.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 9 when executing the program stored in the memory.

12. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 9.

Images