Control method and system of intelligent kitchen electrical equipment

Abstract

The application discloses a control method and system of intelligent kitchen electrical equipment, the method comprises the following steps: a user terminal responds to an equipment adding request, establishes a video call link with a cloud server and collects a current video picture of the intelligent kitchen electrical equipment; the cloud server performs image recognition on the current video picture to obtain an identification result of the intelligent kitchen electrical equipment; the user terminal generates a first virtual identifier corresponding to the intelligent kitchen electrical equipment in the current video picture based on the identification result; the cloud server determines an operation guide corresponding to the intelligent kitchen electrical equipment according to the identification result; the user terminal generates a second virtual identifier corresponding to the intelligent kitchen electrical equipment in the current video picture according to the operation guide; and the user terminal detects an interactive operation in the current video picture and a state change of the intelligent kitchen electrical equipment to obtain a verification result of the interactive operation. The embodiment of the application can adapt to different models of kitchen electrical equipment and provide corresponding guided interactive experience.

Description

Technical Field

[0001] This invention relates to the field of intelligent electrical appliance technology, and in particular to a control method and system for intelligent kitchen appliances. Background Technology

[0002] With the improvement of people's living standards and the promotion and popularization of technologies such as artificial intelligence, more and more traditional lifestyles are gradually changing. The use of home appliances is gradually moving towards intelligence, bringing more convenience to users. At the same time, the functions of various home appliances are becoming more diversified, and the application of smart appliances is increasing. In the current technology, when users use the device for the first time, they usually need to manually add the device through a mobile terminal application. There is a lack of unified operation guidance, which relies on the user's subjective judgment and is prone to problems such as incorrect device model matching and incomplete operation. It is difficult to meet the growing demand for convenient operation of smart devices. Summary of the Invention

[0003] To address the aforementioned problems in the prior art, this invention discloses a control method and system for intelligent kitchen appliances. This system is adaptable to different models of kitchen appliances and provides corresponding guided interactive experiences, improving the efficiency of adding kitchen appliances and the success rate of operation. The technical solution disclosed in this invention is as follows: According to one aspect of the embodiments disclosed in this invention, a control method for an intelligent kitchen appliance is provided, the method comprising: In response to the device addition request, the user terminal establishes a video call link with the cloud server and captures the current video feed of the smart kitchen appliance. The cloud server performs image recognition on the current video frame to obtain the recognition result of the smart kitchen appliance, and sends it to the user terminal; Based on the recognition result, the user terminal generates a first virtual identifier corresponding to the smart kitchen appliance in the current video frame to determine the selection of the smart kitchen appliance and instruct the addition of the smart kitchen appliance; The cloud server determines the corresponding operation instructions for the smart kitchen appliance based on the recognition result and sends them to the user terminal; According to the operation instructions, the user terminal generates a second virtual identifier corresponding to the smart kitchen appliance in the current video screen, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier and control the smart kitchen appliance to perform corresponding operations; The user terminal detects the interactive operations in the current video frame and the status changes of the smart kitchen appliance to obtain the verification result of the interactive operations.

[0004] Optionally, after obtaining the verification result of the interactive operation, the method further includes: If the verification result indicates that the interactive operation was successful or the state of the smart kitchen appliance changes, the user terminal generates a third virtual identifier corresponding to the smart kitchen appliance in the current video frame according to the operation instructions, so as to perform the next interactive operation. If the verification result indicates that the interactive operation has failed or the status of the smart kitchen appliance has not changed, the user terminal increases the intensity of the prompt of the second virtual identifier to enhance the guidance display of the interactive operation location.

[0005] Optionally, the cloud server performs image recognition on the current video frame to obtain the recognition result of the smart kitchen appliance, including: By identifying key frames in the current video frame, the appearance recognition result and feature code recognition result of the smart kitchen appliance are obtained; The appearance recognition result and the feature code recognition result are determined as the recognition result of the smart kitchen appliance.

[0006] Optionally, the user terminal's detection of interactive operations in the current video frame includes: Using gesture recognition or motion capture technology, identify at least one of the following: the user's interaction location, interaction duration, and interaction sequence on the current video screen.

[0007] Optionally, the method further includes: The user terminal sends a role authentication request to the cloud server; The cloud server verifies the user role of the user terminal and determines the permission information of the operation guide based on the user role.

[0008] Optionally, the first virtual identifier includes at least one of a virtual border, a virtual label, or a virtual QR code, and the first virtual identifier is used to identify and mark the selected smart kitchen appliance in the current video frame.

[0009] Optionally, the method further includes: According to the operation instructions, the user terminal generates the second virtual identifier in the current video screen, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier, and control the smart kitchen appliance to perform network distribution operation; The user terminal detects the interactive operations in the current video frame and the network configuration result display information of the smart kitchen appliance in the current video frame to obtain the verification result of the interactive operations.

[0010] Optionally, after obtaining the verification result of the interactive operation, the method further includes: If the verification result indicates that the interactive operation was successful or the network configuration result is displayed in the current video frame, the user terminal updates the display content in the current video frame, prompts that the network configuration operation was successful, and guides the user to complete the network configuration on the user terminal. If the verification result indicates that the interactive operation has failed or the distribution network result display information is not displayed in the current video frame, the user terminal increases the prompt intensity of the second virtual identifier to strengthen the guidance display of the interactive operation location.

[0011] Optionally, the smart kitchen appliances include cooktops, range hoods, dishwashers, steam ovens, and refrigerators.

[0012] According to another aspect of the disclosed embodiments of the present invention, a control system for an intelligent kitchen appliance is provided, the system including a user terminal, a cloud server and an intelligent kitchen appliance, wherein the user terminal, the cloud server and the intelligent kitchen appliance are communicatively connected. The user terminal is used to respond to device addition requests, establish a video call link with the cloud server, and capture the current video feed of the smart kitchen appliance. The cloud server is used to perform image recognition on the current video frame, obtain the recognition result of the smart kitchen appliance, and send it to the user terminal; The user terminal is used to generate a first virtual identifier corresponding to the smart kitchen appliance in the current video frame based on the recognition result, so as to determine the selection of the smart kitchen appliance and instruct the addition of the smart kitchen appliance; The cloud server is used to determine the corresponding operation instructions for the smart kitchen appliance based on the recognition results, and send them to the user terminal; The user terminal is used to generate a second virtual identifier corresponding to the smart kitchen appliance in the current video frame according to the operation instructions, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier, and control the smart kitchen appliance to perform corresponding operations; and to detect the interaction operation in the current video frame and the state change of the smart kitchen appliance to obtain the verification result of the interaction operation.

[0013] The control method and system for intelligent kitchen appliances provided by this invention have the following technical effects: The present invention provides a control method for intelligent kitchen appliances, comprising: a user terminal responding to a device addition request, establishing a video call link with a cloud server, and capturing the current video frame of the intelligent kitchen appliance; then, the cloud server performing image recognition on the current video frame to obtain the recognition result of the intelligent kitchen appliance, and sending it to the user terminal; further, based on the recognition result, the user terminal generating a first virtual identifier corresponding to the intelligent kitchen appliance in the current video frame to confirm the selection of the intelligent kitchen appliance and instruct the addition of the intelligent kitchen appliance; subsequently, the cloud server determining the operation instructions corresponding to the intelligent kitchen appliance based on the recognition result and sending them to the user terminal; then, based on the operation instructions, the user terminal generating a second virtual identifier corresponding to the intelligent kitchen appliance in the current video frame, so as to realize the interaction operation with the intelligent kitchen appliance through the interaction operation with the second virtual identifier, controlling the intelligent kitchen appliance to perform corresponding operations; finally, the user terminal detecting the interaction operation in the current video frame and the state changes of the intelligent kitchen appliance to obtain the verification result of the interaction operation.

[0014] This improves the efficiency and success rate of adding kitchen appliances, lowers the operational threshold, and ensures the reliability of intelligent operation by forming a closed-loop verification based on status detection. It can be adapted to different models of kitchen appliances and provide corresponding guided interactive experience.

[0015] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a control method for an intelligent kitchen appliance according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the feedback processing of an interactive operation verification result according to an exemplary embodiment; Figure 3 This is a schematic diagram illustrating the network control process of an intelligent kitchen appliance according to an exemplary embodiment; Figure 4 This is a schematic diagram illustrating the test control process of an intelligent kitchen appliance according to an exemplary embodiment; Figure 5 This is a schematic diagram illustrating a control system for an intelligent kitchen appliance according to an exemplary embodiment. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions disclosed in this invention, the technical solutions in the disclosed embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention disclosed herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0020] The following describes a control method for an intelligent kitchen appliance according to this application. Figure 1 This is a flowchart illustrating a control method for a smart kitchen appliance according to an exemplary embodiment. This specification provides the operational steps of the method as described in the embodiments or flowchart, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the method can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment) as shown in the embodiments or drawings. Specifically, as... Figure 1 As shown, the above method may include: S101: In response to the device addition request, the user terminal establishes a video call link with the cloud server and captures the current video feed of the smart kitchen appliance.

[0021] In one specific embodiment, the smart kitchen appliance can be any kitchen appliance that requires remote control, specifically including cooktops, range hoods, dishwashers, steam ovens, and refrigerators. The user terminal can be a terminal device running a smart kitchen appliance control app, such as a smartphone or tablet with a built-in camera, display screen, and AR rendering module. The user terminal can be used to perform local operations such as establishing video links, video capture, virtual identifier generation, interaction, and status detection. The cloud server can be a service cluster deployed in the cloud, specifically, it can have a built-in image recognition engine, device knowledge base, and permission management module. It can be used to perform backend computing tasks such as image recognition, operation guide generation and distribution, and communicate with the user terminal via a video call link.

[0022] In one specific embodiment, the device addition request can be an operation request initiated by the user through a user terminal APP to add / bind a smart kitchen appliance. The video call link can be a bidirectional audio and video streaming channel established between the user terminal and the cloud server based on the WebRTC protocol, which can be used to carry the uploading of the current video screen and the distribution of operation instructions. The current video screen can be the image of the smart kitchen appliance captured in real time by the user terminal through the rear camera, which can include visual information such as the device appearance, control panel, screen display content, and device identification (QR code / barcode / SN code), and serves as the input source for image recognition and status detection.

[0023] In practical applications, when a user launches the smart kitchen appliance control app for the first time, needs to add a new kitchen appliance, or when sales / after-sales personnel or development and testing personnel need to configure special working states such as test mode or store mode for the device, a device addition request can be triggered in the app. The user terminal responds to this request by initiating a video call establishment request to the cloud server, establishing a bidirectional real-time video stream transmission channel based on the WebRTC protocol stack. This link carries the device video stream uploaded by the user terminal, as well as the recognition results and AR guidance data sent from the cloud. Optionally, if the smart kitchen appliance itself has video communication capabilities, the user terminal can also directly establish a point-to-point video link with the device, simultaneously synchronizing the link status to the cloud server, where the cloud will complete the subsequent recognition and guidance calculations.

[0024] After the connection is established, the app uses the user's rear camera to guide the user to point the camera at the kitchen appliance to be added, capturing a real-time video stream as the current video frame. During the capture process, the app will guide the user to include the following areas of the device in the frame: the overall appearance of the device, the control panel layout area, the QR code / barcode label area on the device, or the screen display area of ​​devices equipped with color film screens, Linux screens, or Android screens, to ensure the accuracy of subsequent image recognition. Simultaneously, the app will monitor the number of devices in the frame in real time. If multiple similar-looking kitchen appliances are detected simultaneously, a pop-up window will prompt the user to adjust the shooting angle, placing the target device in the center of the frame to avoid ambiguity in subsequent recognition.

[0025] In the embodiments described in this specification, visual features of the device are directly acquired through a video link, replacing the traditional manual operation of screening models and entering serial numbers. This reduces the risk of selection errors caused by similar appearances of similar models from the source, and provides reliable input for subsequent accurate identification and binding of the device.

[0026] S103: The cloud server performs image recognition on the current video frame, obtains the recognition results of the smart kitchen appliances, and sends them to the user terminal.

[0027] In one specific embodiment, image recognition can be a visual analysis operation performed by a cloud server on key frames of the current video screen, which may include appearance recognition (identifying device shape, panel layout, and screen type) and feature code extraction (identifying device identification and screen display codes via OCR). The recognition result can be information that uniquely identifies the target smart kitchen appliance output by image recognition, which may include at least the device model, and may also include associated information such as the corresponding network configuration protocol and firmware version.

[0028] Optionally, the cloud server may perform image recognition on the current video frame to obtain the recognition results of the smart kitchen appliances, which may include: By identifying key frames in the current video frame, the appearance recognition results and feature code recognition results of the smart kitchen appliances are obtained; The appearance recognition result and the feature code recognition result are determined as the recognition result of the smart kitchen appliance.

[0029] In one specific embodiment, the cloud server can receive real-time video streams uploaded by user terminals through a video stream processing service module. Following a preset time interval or scene change detection algorithm (such as inter-frame difference), it extracts keyframes containing complete device features from the video stream and inputs them into the image recognition engine module for parallel analysis. Specifically, keyframes can be representative frames extracted from continuously transmitted real-time video streams for image analysis. The selection of keyframes can be based on preset time intervals or the magnitude of image changes, aiming to reduce the cloud computing load while ensuring recognition accuracy. Appearance recognition results can be obtained by analyzing the device's visual features (shape, panel layout, screen type) using a deep learning model; specifically, this can include the device's series, model number, etc. Feature code recognition results can be precise characters or encoded information (such as serial number, model string, QR code content) extracted from the device's sticker or screen display content using OCR technology or code parsing algorithms.

[0030] Specifically, appearance recognition can load a pre-trained deep learning model based on CNN (Convolutional Neural Network) into the image recognition engine to perform appearance recognition on keyframes. This can include: recognizing the overall shape and outline of the device, the layout of the control panel (such as button arrangement and knob position), and the screen type (such as color filter screen, Linux screen, or Android screen), and outputting the appearance recognition result. Feature code recognition can call the OCR (Optical Character Recognition) submodule and the QR code / barcode parsing submodule through the image recognition engine to perform feature code recognition on keyframes. This can include: locating the sticker area on the device body and recognizing the model string and serial number (SN); or parsing the dynamic verification code or network configuration QR code currently displayed on the device screen, which is equivalent to the device's unique identifier, i.e., the feature code recognition result.

[0031] S105: Based on the recognition result, the user terminal generates a first virtual identifier corresponding to the smart kitchen appliance in the current video frame to confirm the selection of the smart kitchen appliance and instruct the user to add the smart kitchen appliance.

[0032] In one specific embodiment, the first virtual identifier can be a visual locking element superimposed on the current video screen by the user terminal through augmented reality (AR) technology. The form of the identifier can include a highlighted border, a virtual label, a positioning anchor point, and a virtual QR code. It can be used to clearly lock the target device that the user is pointing at in a multi-device coexistence scenario, eliminate selection ambiguity, and identify and mark the selected smart kitchen appliance in the current video screen.

[0033] S107: The cloud server determines the corresponding operation instructions for the smart kitchen appliances based on the recognition results and sends them to the user terminal.

[0034] In one specific embodiment, the operation guide can be a standard operating procedure (SOP) retrieved by the cloud server from a pre-stored device operation knowledge base based on the recognition result. It may include specific operation rules (such as button position, press duration, menu path) required for the target device to enter user mode, test mode, store mode, etc.

[0035] In practical applications, after receiving the recognition results (which at least include the model and screen type of the smart kitchen appliance) from the image recognition module, the cloud server uses these results as an index to query the device knowledge base service module deployed in the cloud. This module pre-stores the operation knowledge base corresponding to the kitchen appliance, establishing a mapping between device models and standardized operating procedures (SOPs). Specifically, for each specific device model, the knowledge base stores the following data: the physical button combinations, touch operation sequences, press duration thresholds, and corresponding screen menu hierarchy paths required to enter user mode, test mode, store mode, etc. Additionally, the knowledge base maintains a button coordinate mapping table and a screen menu hierarchy structure, which can be used to accurately determine the overlay position of AR virtual icons in the video frame.

[0036] Subsequently, before finalizing the operation instructions, the cloud server's permission management service module first verifies the user's role permissions based on the current user's login credentials. If the verification result indicates a regular user, only basic mode (such as network configuration mode) instructions are issued; if the verification result indicates a sales / after-sales personnel or developer (requiring account permission verification or dynamic verification code verification), then a combination key sequence or deep menu path instruction containing high-privilege operations such as test mode and store mode is issued. Based on the device model in the recognition result and the verified user role, the cloud server retrieves matching SOP data from the knowledge base and generates a structured dynamic operation instruction containing virtual button positions, dynamic arrow trajectories, countdown parameters, and voice prompt text. The cloud server then sends this dynamic operation instruction to the user terminal to trigger subsequent AR rendering and interactive guidance.

[0037] S109: According to the operation instructions, the user terminal generates a second virtual identifier corresponding to the smart kitchen appliance in the current video screen, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier and control the smart kitchen appliance to perform corresponding operations.

[0038] In one specific embodiment, the second virtual identifier can be an operation guidance element superimposed on the current video screen by the user terminal based on the operation instructions using AR technology. Its form can include flashing dots, dynamic arrows, hand icons, key combination numbers, menu path indicators, etc., and can be used to intuitively prompt the user to perform corresponding interactive operations. The interactive operation can be the actual physical operation performed by the user on the control panel of the smart kitchen appliance following the prompts of the second virtual identifier. This can include, but is not limited to, pressing physical buttons, clicking on the touchscreen, long-pressing, key combination operations, etc., and serves as input to trigger changes in the device's state.

[0039] In practical applications, after receiving dynamic operation instructions from the cloud server, the AR rendering module on the user terminal first parses the core parameters carried in the instructions. These parameters may include the target operation location coordinates, the operation type (press / click / long press / combined operation), the operation duration threshold, and the permission level corresponding to the current user role. The target operation location coordinates are obtained from a button coordinate mapping table or screen menu hierarchy sent from the cloud, based on the device model identified above, ensuring a one-to-one correspondence between the location and the actual physical buttons and touch areas of the device.

[0040] Based on the device's spatial location locked by the first virtual identifier, the user terminal uses augmented reality spatial anchoring technology to precisely overlay the second virtual identifier onto the location of the corresponding actual button or screen menu item in the current video frame. Even if the camera angle shifts slightly, the virtual identifier will adjust synchronously with the device position to avoid misalignment of guidance.

[0041] Optionally, the representation of the second virtual identifier can be dynamically adapted to the operation scenario and user role, specifically including: In a typical user network configuration scenario: the second virtual identifier can be a flashing dot (e.g., with a countdown timer) or an animated hand gesture, anchored at the actual location of the network configuration trigger button, prompting the user to "Please press and hold the XX button here." The user terminal's gesture / action recognition module captures the user's finger pressing position and movement in real time. If it detects that the user has completed a pressing operation that meets the required duration at the actual location anchored to the virtual identifier, the initial interaction is considered successful, and the guidance automatically switches to the next prompt (e.g., "Please wait for the device to send a hotspot signal").

[0042] In high-privilege scenarios for after-sales / development testing: After sales or development personnel can obtain advanced permissions through authentication, the second virtual identifier is a numerical sequence number marked according to the operation order (such as 1, 2, 3), which is anchored to the physical button positions corresponding to the combination key, prompting the user to press in sequence; if the target device is equipped with a color film screen, Linux screen, or Android screen, the second virtual identifier is a dynamic arrow, anchored to the corresponding option position of engineering mode or test mode in the current menu on the screen, prompting the user "Please select XX mode under this menu".

[0043] In the store mode scenario: the second virtual identifier can be a flashing menu path indicator line, anchored to the corresponding position of the screen menu or physical button, prompting the user to trigger the store display mode with one click, without having to consult the operation manual or memorize complex paths.

[0044] In the embodiments described in this specification, regardless of whether the kitchen appliance has a physical button panel, a touch screen, or a smart operating system screen, the user's interaction method is uniform. Users simply need to follow the flashing second virtual identifier in the video image and click or press its anchored location to complete the operation; there is no need to understand the differentiated operating logic of different devices. The user terminal synchronizes the preliminary judgment result of the interaction operation to the subsequent verification steps, where the cloud or local system further detects changes in the device status to confirm whether the operation has ultimately taken effect.

[0045] S111: The user terminal detects the interactive operations and status changes of the smart kitchen appliances in the current video frame, and obtains the verification results of the interactive operations.

[0046] In one specific embodiment, the state change can be a visually identifiable change in the state of the smart kitchen appliance after responding to an interactive operation, including switching of screen display content (such as entering the network configuration interface, displaying a hotspot icon, or popping up a test menu) and changes in indicator light status (such as switching from constant on to flashing). The verification result can be a judgment conclusion drawn by the user terminal through real-time analysis of the current video screen and comparison of the interactive operation with the state change. It can be used to confirm whether the user operation is correct and whether the device has successfully entered the target mode, thereby forming a closed loop of operation-feedback-re-guidance.

[0047] Optionally, the user terminal's detection of interactive operations in the current video frame may include: Using gesture recognition or motion capture technology, identify at least one of the following: the user's interaction location, interaction duration, and interaction sequence on the current video screen.

[0048] Specifically, gesture recognition technology can be used to identify touch gestures such as clicks, long presses, and swipes performed by users on the user terminal's touchscreen in response to AR-overlaid second virtual icons. Motion capture technology can be used to capture the user's finger touch trajectories and pressing actions on the user terminal's touchscreen, collecting finger contact coordinates, pressing duration, and action sequences through touchscreen drivers. The interaction location can be the user's finger pressing coordinates on the user terminal's touchscreen, mapped to the corresponding pixel position of the second virtual icon in the current video frame. The interaction duration can be the duration for which the user presses the second virtual icon on the user terminal's touchscreen, which can be used to verify whether the virtual operation meets the required pressing threshold (e.g., a 3-second long press). The interaction sequence can be the trigger timestamp sequence of multiple second virtual icons clicked sequentially by the user, which can be used to verify the logical correctness of high-privilege combined virtual operations.

[0049] In practical applications, the AR rendering module of the user terminal has precisely overlaid a second virtual identifier (such as a flashing dot, a key combination number, or a menu path indicator line) onto the current video screen using spatial anchoring technology, corresponding to the target operation positions of the physical kitchen appliance, such as the network connection button, key combination button, and on-screen menu item. Users do not need to approach or touch the physical kitchen appliance; they only need to click / press the corresponding position on the user terminal's touchscreen to trigger the operation.

[0050] The user terminal's touchscreen driver captures the user's finger touch events in real time, and the gesture / motion recognition module performs the following recognition: Interactive position recognition: The system maps the finger press coordinates captured by the touchscreen to the pixel coordinate system of the current video screen, determines whether the coordinates fall within the preset anchoring area of ​​the second virtual identifier, and confirms whether the user has clicked the target virtual operation position required by the guidance. For example, if the user clicks the flashing dot anchored at the "Network Configuration Button" position on the screen, the system recognizes it as selecting the network configuration trigger operation.

[0051] Interaction duration recognition: The system starts timing from the moment a finger touches the second virtual identifier area on the touchscreen and ends when the finger leaves, calculating the duration of the press. For example, if a user long-presses the "Test Mode" virtual identifier on the screen, the system identifies whether the press duration reaches the threshold required by the operation guidelines, distinguishing between a click and a long press.

[0052] Interaction sequence recognition: For high-privilege combined operations (such as the key combination sequence triggered by after-sales personnel in test mode), the system records the timestamps of multiple second virtual icons clicked in sequence by the user, and identifies whether the click order conforms to the 1, 2, 3 sequence logic indicated by the guidance. For example, if the user needs to click the menu button marked "1", the temperature button marked "2", and the fan speed button marked "3" on the screen in sequence, the system will identify whether the operation sequence meets the requirements.

[0053] The above three types of recognition results can be output individually or in combination. In ordinary user network configuration scenarios, only the interaction location and duration need to be recognized, while in after-sales / development high-privilege scenarios, all three types of parameters can be recognized simultaneously. After recognition, the user terminal compares the interaction parameters with the preset requirements of the operation guide, generates a preliminary verification result, and synchronizes it to the subsequent verification steps. The cloud server or user terminal cross-verifies the verification result with the status changes of the smart kitchen appliance (such as indicator light flashing and screen menu jumps) to confirm whether the virtual operation is effective.

[0054] Optionally, the above method may also include: The user terminal sends a role authentication request to the cloud server; The cloud server verifies the user's role on the user terminal and determines the permission information for the operation guide based on the user's role.

[0055] In one specific embodiment, the role authentication request can be a request message initiated by the user terminal to the cloud server to verify the identity and permissions of the operating user. It can carry implicit credentials (such as a login token) or explicit verification information (such as a work badge QR code or dynamic verification code) and is the triggering condition for cloud permission verification. User roles can be role categories divided according to the identity attributes and responsibilities of the operating user. Specifically, they can include ordinary users: terminal users of smart kitchen appliances, who only have the operation permissions of device binding and basic network configuration; after-sales personnel: professionals responsible for equipment repair and store debugging, who have the operation permissions of after-sales scenarios such as test mode and store mode; and developers: technicians responsible for equipment research and development and testing, who have the operation permissions of research and development scenarios such as underlying debugging and firmware configuration.

[0056] In practical applications, the user terminal first determines whether role authentication is required based on the attributes of the currently logged-in account or the user's selected operation mode (such as normal network configuration, test mode, or store mode). If the user is a regular user and only requests basic network configuration guidance, the user terminal automatically sends an implicit authentication request to the cloud, carrying the valid identity token obtained during login. If the user chooses to enter higher-privilege operations such as test mode or store mode, or if the current terminal is an after-sales terminal or development terminal, the user terminal can guide the user to complete identity verification by scanning the employee badge QR code, entering a dynamic verification code, or verifying the enterprise account password, generating an explicit role authentication request and sending it to the cloud.

[0057] Based on the verified user roles, the cloud server determines the corresponding permission information as constraints for subsequent retrieval of operation guidance: Ordinary users can only access the standard operating procedures (SOPs) in network configuration mode; AR guidance only displays single-step operation prompts and cannot trigger engineering modes. After-sales personnel can access the SOPs in test and store modes; AR guidance can display detailed guidance such as key combination numbers and menu paths. Developers can access all operation SOPs; AR guidance can display advanced guidance such as underlying parameter configurations and debugging commands. Optionally, for operation requests from high-privilege roles, the cloud server can synchronously record operation log audit information, including the operation user ID, target device model, requested operation type, and request timestamp, which can be used for subsequent operation traceability.

[0058] If role verification fails, the cloud server returns an insufficient permissions message to the user terminal, and the AR guidance layer on the user terminal displays "No permission to access this mode, please contact the administrator," terminating the subsequent operation guidance delivery process. If verification succeeds, the cloud server retrieves the corresponding operation guidance based on the device identification result and permission information.

[0059] Optional, such as Figure 2As shown, after obtaining the verification result of the interactive operation, the above method may further include: S201: If the verification result indicates that the interactive operation was successful or the status of the smart kitchen appliance has changed, the user terminal generates a third virtual identifier corresponding to the smart kitchen appliance in the current video screen according to the operation instructions, so as to carry out the next interactive operation.

[0060] In one specific embodiment, the third virtual identifier can be an AR guidance element superimposed on the current video screen after the user terminal verifies the success of the current interactive operation, based on the next step of the operation guidance. It forms a full-process guidance chain with the first virtual identifier (locking the target device) and the second virtual identifier (guiding the current operation). The specific forms may include a confirmation icon, a flashing indicator of the next operation position, a process completion prompt, etc., to instruct the user to perform the next round of interactive operation.

[0061] S203: If the verification result indicates that the interactive operation has failed or the status of the smart kitchen appliance has not changed, the user terminal increases the intensity of the prompt of the second virtual identifier to strengthen the guidance display of the interactive operation location.

[0062] In one specific embodiment, the prompt intensity can be the perceptibility of the second virtual icon in the visual and auditory dimensions. Specifically, it can be enhanced by adjusting the icon color, flashing frequency, size, overlay animation (such as shaking, scaling), and the volume and content of the accompanying voice prompts. This can be used to strengthen the guidance effect of the correct operation position and reduce the probability of user misoperation.

[0063] In practical applications, when the identified user interaction location, duration, and sequence meet the requirements of the operation guidelines, or when an expected change in the status of the smart kitchen appliance is detected (e.g., the indicator light switches to flashing mode after the network connection button is pressed, the screen enters the engineering menu after the test combination button is pressed, or the screen starts a loop demonstration after the store mode button is pressed), the user terminal can overlay a third virtual identifier onto the current video screen based on the next step requirement of the operation guidelines. The presentation of the third virtual identifier can be dynamically adapted according to the scenario: For typical user network configuration scenarios: If the current operation is a single-step network configuration, the third virtual identifier can be a confirmation icon (such as a checkmark icon) and the text message "Network configuration successful, please connect to the device hotspot". If the operation is a multi-step network configuration (such as first pressing and holding the network configuration button, and then selecting a WiFi network), the third virtual identifier can be a flashing dot indicating the next operation position, while the original second virtual identifier turns gray, prompting the user to perform the next operation.

[0064] For after-sales / development testing scenarios: For key combination operations, the completed step number (e.g., "1") is grayed out, and the third virtual identifier can be a flashing indicator for the next step number (e.g., "2"), anchored to the next target key position. For menu operations on color screens / Linux screens / Android screens, the third virtual identifier can be a dynamic arrow, anchored to the "Engineering Mode" confirmation item position after the screen jumps.

[0065] For the store mode scenario: the third virtual identifier can be a floating label that says "Demo is started", while also overlaying the parameter description of the store display mode (such as "Loop demo duration: 30 seconds").

[0066] Optionally, if the user interaction location deviates from the target area, the duration is insufficient, the sequence is incorrect, or the device status does not change (e.g., the indicator light does not flash after the button is pressed, or the screen does not change), the user terminal can perform the following operations: Enhance the intensity of the second virtual identifier prompt: Strengthen guidance through multi-dimensional visual and auditory cues. Visually, the color of the second virtual identifier can be enhanced / changed (e.g., switched to warning red), a shaking animation can be overlaid, it can be enlarged, and the flashing frequency can be increased. Auditorily, voice prompts can be played simultaneously (e.g., "Incorrect operation, please try again"), accompanied by text pop-ups explaining the error type (e.g., "Please press and hold for 3 seconds," "Please press in the correct order"), etc.

[0067] Auxiliary verification prompts: If consecutive operations fail, the user terminal can display an auxiliary prompt: "Please confirm that you are aligned with the target device, or adjust the lighting and try again." This prevents users from failing due to selecting the wrong device or obstructed lighting in multi-device scenarios. For high-privilege testing / development scenarios, the failure prompt only indicates "Operation error," without revealing specific key combinations or menu paths, thus meeting the security control requirements for sensitive operations.

[0068] In the above embodiments, a complete closed loop is formed by dynamically updating AR guidance. Users can intuitively understand the operation results and adjust their behavior without consulting paper / electronic manuals. This reduces the cognitive load of ordinary users and ensures the security and ease of use of high-privilege operations.

[0069] Optionally, the above method may also include: According to the operation instructions, the user terminal generates a second virtual identifier in the current video screen, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier, and control the smart kitchen appliance to perform network distribution operation; The user terminal detects the interactive operations in the current video frame and the network configuration results displayed by the smart kitchen appliances in the current video frame to obtain the verification results of the interactive operations.

[0070] In one specific embodiment, the network configuration operation can be a pre-operation for ordinary users to configure wireless network connections for smart kitchen appliances. Through virtual interaction, the device is triggered to enter network configuration mode, causing it to emit a WiFi hotspot or display a network configuration QR code, awaiting user terminal access. The network configuration result display information can be visually identifiable status characteristics of the smart kitchen appliance after successfully entering network configuration mode. Specifically, this may include a network hotspot icon, a network configuration QR code, a "waiting to connect" message, a dynamic verification code, and a preset flashing pattern for the network configuration indicator light displayed on the screen.

[0071] Optionally, after obtaining the verification results of the interactive operation, the above method may further include: If the verification result indicates that the interactive operation was successful or the current video screen displays network configuration result information, the user terminal updates the display content in the current video screen, prompts that the network configuration operation was successful, and guides the user to complete the network configuration on the user terminal. If the verification result indicates that the interactive operation failed or the distribution network result display information is not shown in the current video screen, the user terminal increases the prompt intensity of the second virtual identifier to strengthen the guidance display of the interactive operation location.

[0072] In practical applications, such as Figure 3 As shown, in the network configuration mode guidance scenario, the cloud sends AR button guidance (blinking and highlighting the target button) based on the identified device model; the user operates the kitchen appliance buttons according to the guidance; the device emits a network configuration hotspot; the user terminal collects and uploads the device status screen; the cloud identifies the hotspot icon or screen status change; confirms successful network configuration and prompts the user. Figure 4 As shown, for after-sales or remote guidance scenarios for developers, the after-sales / development terminal initiates an assistance request; the cloud verifies the technician's role permissions (high-permission mode); AR key combination guidance for test mode or store mode is issued; the user operates the device according to the guidance; the device enters the target mode; the user terminal collects and uploads the device screen image; the cloud identifies the status display of test mode, etc.; and returns a confirmation of successful operation.

[0073] In practical applications, the user terminal parses the network configuration operation instructions sent from the cloud (the instructions are obtained from a pre-stored device knowledge base and include parameters such as the coordinates of the network configuration trigger key of the kitchen appliance, the press duration threshold, and network configuration status characteristics). Based on the target device's spatial location locked by the aforementioned first virtual identifier, augmented reality spatial anchoring technology is used to precisely overlay the second virtual identifier onto the position of the corresponding physical device's network configuration trigger key (such as the "Menu" key, "Fan Speed" key, or "Network Configuration" dedicated key) in the current video frame. Optionally, the second virtual identifier can be a flashing dot with a countdown circle, accompanied by a hand animation, and can also be overlaid with the text prompt "Please press and hold the network configuration key here." Even if the camera angle shifts slightly, the virtual identifier will adjust synchronously with the target device to avoid misalignment of the instructions.

[0074] The user terminal's gesture / action recognition module identifies interaction parameters in real time, including interaction position (determining whether the touch coordinates of the user's finger press fall into the preset anchoring area of ​​the second virtual icon) and interaction duration (calculating whether the duration of the finger press reaches the required duration threshold (e.g., 3 seconds).

[0075] If the interaction location hits the second virtual identifier area, the interaction duration is greater than or equal to the duration threshold, and any distribution network result display information is detected within the preset timeout period (e.g., 10 seconds), the verification result is determined to be a successful interaction operation. If the interaction location deviates from the anchor area of ​​the second virtual identifier, or the interaction duration is less than the duration threshold, or no distribution network result display information is detected after the preset timeout period, the verification result is determined to be a failed interaction operation.

[0076] Upon successful verification, the user terminal updates the displayed content in the current video frame, generating a third virtual identifier: this can display a confirmation icon, the text "Network configuration successful, please connect to the device hotspot," and a network configuration guidance pop-up window. This guides the user to select the Wi-Fi hotspot emitted by the device and enter their home Wi-Fi account and password within the user terminal's app interface, completing the final network configuration. The user terminal simultaneously uploads the successful network configuration progress to the cloud, eliminating the need to repeat the device identification process when the user reopens the app or switches devices.

[0077] In the event of verification failure, the user terminal can enhance the alert intensity of the second virtual identifier. For example, visually, the color of the second virtual identifier can be changed to warning red, and a shaking animation can be superimposed, the identifier can be enlarged, and the flashing frequency can be increased. Auditorily, voice prompts can be played simultaneously (such as "Operation error, please press and hold here for 3 seconds" or "Device not responding, please check the power and try again"). If the operation fails repeatedly, an additional auxiliary prompt will pop up: "Please adjust the camera angle to ensure that the device is fully in the frame."

[0078] As can be seen from the technical solutions provided in the embodiments of this specification above, in response to a device addition request, the user terminal establishes a video call link with the cloud server and captures the current video frame of the smart kitchen appliance. Then, the cloud server performs image recognition on the current video frame to obtain the recognition result of the smart kitchen appliance and sends it to the user terminal. Further, based on the recognition result, the user terminal generates a first virtual identifier corresponding to the smart kitchen appliance in the current video frame to confirm the selection of the smart kitchen appliance and instruct the user to add it. Then, the cloud server determines the operation instructions corresponding to the smart kitchen appliance based on the recognition result and sends them to the user terminal. Then, based on the operation instructions, the user terminal generates a second virtual identifier corresponding to the smart kitchen appliance in the current video frame to achieve interactive operation with the smart kitchen appliance through interaction with the second virtual identifier, controlling the smart kitchen appliance to perform corresponding operations. Finally, the user terminal detects the interactive operation in the current video frame and the state changes of the smart kitchen appliance to obtain the verification result of the interactive operation.

[0079] This improves the efficiency and success rate of adding kitchen appliances, lowers the operational threshold, and ensures the reliability of intelligent operation by forming a closed-loop verification based on status detection. It can be adapted to different models of kitchen appliances and provide corresponding guided interactive experience.

[0080] The present invention also provides a control system for intelligent kitchen appliances, the system including a user terminal, a cloud server and intelligent kitchen appliances, and the user terminal, the cloud server and the intelligent kitchen appliances are connected in communication. The user terminal is used to respond to device addition requests, establish a video call link with the cloud server, and capture the current video feed of the smart kitchen appliance. The cloud server is used to perform image recognition on the current video frame, obtain the recognition results of the smart kitchen appliances, and send them to the user terminal; The user terminal is used to generate a first virtual identifier for the smart kitchen appliance in the current video frame based on the recognition result, so as to determine the selection of the smart kitchen appliance and instruct the addition of the smart kitchen appliance; The cloud server is used to determine the corresponding operation instructions for the smart kitchen appliances based on the recognition results and send them to the user terminal; The user terminal is used to generate a second virtual identifier corresponding to the smart kitchen appliance in the current video frame according to the operation instructions, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier, and control the smart kitchen appliance to perform corresponding operations; and to detect the interaction operation and the state change of the smart kitchen appliance in the current video frame, and obtain the verification result of the interaction operation.

[0081] Optionally, the mobile terminal can also be used to generate a third virtual identifier corresponding to the smart kitchen appliance in the current video frame according to the operation instructions, so as to carry out the next interactive operation, when the verification result indicates that the interactive operation is successful or the status of the smart kitchen appliance changes; and to enhance the prompt intensity of the second virtual identifier, so as to strengthen the guidance display of the interactive operation location, when the verification result indicates that the interactive operation fails or the status of the smart kitchen appliance does not change.

[0082] Optionally, the cloud server can also be used to identify key frames in the current video frame to obtain the appearance recognition result and feature code recognition result of the smart kitchen appliance; and to determine the appearance recognition result and feature code recognition result as the recognition result of the smart kitchen appliance.

[0083] Optionally, the mobile terminal can also be used to identify at least one of the following: the user's interaction location, interaction duration, and interaction sequence on the current video screen, through gesture recognition or motion capture technology.

[0084] Optionally, the mobile terminal can also be used to send role authentication requests to the cloud server; The cloud server can also be used to verify the user role of the user terminal and determine the permission information of the operation guide based on the user role.

[0085] Optionally, the first virtual identifier includes at least one of a virtual border, a virtual label, or a virtual QR code, and the first virtual identifier is used to identify and mark the selected smart kitchen appliance in the current video frame.

[0086] Optionally, the user terminal can also be used to generate a second virtual identifier in the current video frame according to the operation instructions, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier, and control the smart kitchen appliance to perform the network distribution operation; and to detect the interaction operation in the current video frame and the network distribution result display information of the smart kitchen appliance in the current video frame, so as to obtain the verification result of the interaction operation.

[0087] Optionally, the user terminal can also be used to update the displayed content in the current video screen when the verification result indicates that the interactive operation was successful or when the current video screen displays network configuration result information, prompting that the network configuration operation was successful and guiding the user to complete the network configuration on the user terminal; and to enhance the prompt intensity of the second virtual identifier when the verification result indicates that the interactive operation failed or when the current video screen does not display network configuration result information, so as to strengthen the guidance display of the location of the interactive operation.

[0088] Optional smart kitchen appliances include cooktops, range hoods, dishwashers, steam ovens, and refrigerators.

[0089] Optional, such as Figure 5 As shown, the user terminal can include clients for users of the smart cooking equipment, clients for sales personnel, and clients for developers. In practical applications, each type of user can obtain different levels of permissions after logging in and authenticating on the client. For example, developers can enter engineering mode or test mode, while users and sales personnel do not have such permissions. The user terminal has a built-in camera for capturing video footage from the control panel of the kitchen appliance. The cloud server cluster can include a video stream processing service module, an image recognition engine module, a device knowledge base service module, and a permission management service module. The smart kitchen appliance includes a control panel (physical buttons or a touchscreen) and a communication module, supporting WiFi network configuration and remote control command reception.

[0090] Regarding the control system of the intelligent kitchen appliance in the above embodiments, the specific way in which each module performs operations has been described in detail in the embodiments of the control method of the intelligent kitchen appliance, and will not be elaborated here.

[0091] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0092] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles disclosed herein and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0093] It should be understood that the present invention is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is limited only by the appended claims.

Claims

1. A control method for intelligent kitchen appliances, characterized in that, The method includes: In response to the device addition request, the user terminal establishes a video call link with the cloud server and captures the current video feed of the smart kitchen appliance. The cloud server performs image recognition on the current video frame to obtain the recognition result of the smart kitchen appliance, and sends it to the user terminal; Based on the recognition result, the user terminal generates a first virtual identifier corresponding to the smart kitchen appliance in the current video frame to determine the selection of the smart kitchen appliance and instruct the addition of the smart kitchen appliance; The cloud server determines the corresponding operation instructions for the smart kitchen appliance based on the recognition result and sends them to the user terminal; According to the operation instructions, the user terminal generates a second virtual identifier corresponding to the smart kitchen appliance in the current video screen, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier and control the smart kitchen appliance to perform corresponding operations; The user terminal detects the interactive operations in the current video frame and the status changes of the smart kitchen appliance to obtain the verification result of the interactive operations.

2. The method according to claim 1, characterized in that, After obtaining the verification result of the interactive operation, the method further includes: If the verification result indicates that the interactive operation was successful or the state of the smart kitchen appliance changes, the user terminal generates a third virtual identifier corresponding to the smart kitchen appliance in the current video frame according to the operation instructions, so as to perform the next interactive operation. If the verification result indicates that the interactive operation has failed or the status of the smart kitchen appliance has not changed, the user terminal increases the intensity of the prompt of the second virtual identifier to enhance the guidance display of the interactive operation location.

3. The method according to claim 1, characterized in that, The cloud server performs image recognition on the current video frame to obtain the recognition result of the smart kitchen appliance, including: By identifying key frames in the current video frame, the appearance recognition result and feature code recognition result of the smart kitchen appliance are obtained; The appearance recognition result and the feature code recognition result are determined as the recognition result of the smart kitchen appliance.

4. The method according to claim 1, characterized in that, The user terminal detects interactive operations in the current video frame, including: Using gesture recognition or motion capture technology, identify at least one of the following: the user's interaction location, interaction duration, and interaction sequence on the current video screen.

5. The method according to claim 1, characterized in that, The method further includes: The user terminal sends a role authentication request to the cloud server; The cloud server verifies the user role of the user terminal and determines the permission information of the operation guide based on the user role.

6. The method according to claim 1, characterized in that, The first virtual identifier includes at least one of a virtual border, a virtual label, or a virtual QR code, and is used to identify and mark the selected smart kitchen appliance in the current video frame.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: According to the operation instructions, the user terminal generates the second virtual identifier in the current video screen, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier, and control the smart kitchen appliance to perform network distribution operation; The user terminal detects the interactive operations in the current video frame and the network configuration result display information of the smart kitchen appliance in the current video frame to obtain the verification result of the interactive operations.

8. The method according to claim 7, characterized in that, After obtaining the verification result of the interactive operation, the method further includes: If the verification result indicates that the interactive operation was successful or the network configuration result is displayed in the current video frame, the user terminal updates the display content in the current video frame, prompts that the network configuration operation was successful, and guides the user to complete the network configuration on the user terminal. If the verification result indicates that the interactive operation has failed or the distribution network result display information is not displayed in the current video frame, the user terminal increases the prompt intensity of the second virtual identifier to strengthen the guidance display of the interactive operation location.

9. The method according to any one of claims 1 to 6, characterized in that, The smart kitchen appliances include cooktops, range hoods, dishwashers, steam ovens, and refrigerators.

10. A control system for an intelligent kitchen appliance, characterized in that, The system includes a user terminal, a cloud server, and smart kitchen appliances, and the user terminal, the cloud server, and the smart kitchen appliances are connected in communication. The user terminal is used to respond to device addition requests, establish a video call link with the cloud server, and capture the current video feed of the smart kitchen appliance. The cloud server is used to perform image recognition on the current video frame, obtain the recognition result of the smart kitchen appliance, and send it to the user terminal; The user terminal is used to generate a first virtual identifier corresponding to the smart kitchen appliance in the current video frame based on the recognition result, so as to determine the selection of the smart kitchen appliance and instruct the addition of the smart kitchen appliance; The cloud server is used to determine the corresponding operation instructions for the smart kitchen appliance based on the recognition results, and send them to the user terminal; The user terminal is used to generate a second virtual identifier corresponding to the smart kitchen appliance in the current video frame according to the operation instructions, so as to realize the interaction operation with the smart kitchen appliance through the interaction operation with the second virtual identifier and control the smart kitchen appliance to perform corresponding operations; And to detect interactive operations in the current video frame and state changes of the smart kitchen appliance, and obtain the verification result of the interactive operation.

Images