A smart robot for home kitchens and its control method
This smart kitchen robot, controlled by both voice and gestures, combines real-time video streaming and voice recognition technology to solve the problems of high cost and difficult operation of existing smart kitchen equipment. It achieves low-cost, personalized intelligent control, making it suitable for different user groups.
Patent Information
- Application Number
- CN202311423884.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-30
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-10-30
AI Technical Summary
Existing smart kitchen equipment is expensive, difficult for special groups such as the elderly and teenagers to operate, and lacks personalized and integrated intelligent systems, failing to meet the special needs of different users.
This smart kitchen robot uses both voice and gesture control. It combines real-time video streaming and voice recognition technology, and uses a four-level cascaded processing algorithm and MediaPipe algorithm for gesture recognition. It then connects to a smart cloud platform for personalized recipe customization.
It provides a low-cost, efficient, and convenient intelligent experience, suitable for different age groups, meets personalized needs, and achieves comprehensive intelligent control.
Smart Images

Figure CN117289803B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of kitchen control technology, and in particular to a smart robot for home kitchens and its control method. Background Technology
[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.
[0003] With the development of technology, more and more smart devices have been or will be integrated into people's lives, making life more convenient and comfortable. The kitchen is no exception, and a large number of smart kitchen devices have emerged.
[0004] Existing smart kitchen equipment, one type being the integrated smart kitchen, is a fully automated system utilizing the Internet of Things (IoT). While this solution allows for control of the entire kitchen via a connected smartphone, the equipment cost is prohibitively high for most people. Furthermore, overly intelligent operating methods are not user-friendly for special groups such as the elderly, who may be unfamiliar with or find it inconvenient to use smartphones and similar devices.
[0005] Another approach is a decentralized smart home ecosystem, such as smart ovens and smart refrigerators. While this solution involves the intelligent transformation of specific kitchen appliances, it also suffers from high costs. Furthermore, individual appliances cannot form a complete intelligent system, lacking overall cohesion and integration. Additionally, this approach typically lacks personalization, failing to meet the specific needs and preferences of different users. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides a smart home kitchen robot, a control method, an electronic device, and a computer-readable storage medium. It is highly practical, suitable for different age groups, and has a high accuracy rate in gesture recognition.
[0007] In a first aspect, the present invention provides a method for controlling a smart robot in a household kitchen; A method for controlling a smart robot in a home kitchen includes: Acquire and recognize real-time voice or video streams, determine control parameters, and control a smart home kitchen robot based on these parameters. Acquiring and identifying real-time video streams includes: Acquire real-time video stream, process the real-time video stream, and obtain key gesture points and their coordinate information; A four-level cascaded processing algorithm is used to process the key points of the gesture and their coordinate information to obtain the gesture recognition result; The gesture recognition results are compared with the preset meanings of dynamic gestures to determine the control parameters.
[0008] Furthermore, the processing of gesture key points and their coordinate information through a four-level cascaded processing algorithm includes: Based on the key points of the gesture and their coordinate information, obtain the number of protruding key points of the fingertips; based on the number of protruding points, determine the number of fingers that extend. Based on the number of extended fingers, key finger points and their coordinates, obtain the angle information between key finger points, and determine the degree of bending of a single finger joint based on the angle information; Based on the degree of bending of individual finger joints, the angles between different fingers are obtained to determine the preliminary gesture recognition results; Based on the key points of the fingertips and their coordinate information, calculate the ratio of the distances between the key points of the fingertips; based on the distance ratio and the preliminary gesture recognition results, obtain the final gesture recognition result.
[0009] Furthermore, acquiring and recognizing real-time speech specifically involves: Acquire real-time voice and compare its features with a preset wake-up word library to determine the control mode; The control parameters are determined based on the control mode and real-time voice prompts.
[0010] Preferably, the step of acquiring real-time voice and comparing its features with a preset wake-up word library to determine the control mode includes: If the real-time voice contains a preset first wake-up word, control parameters are generated to establish interaction with the intelligent chatbot platform or the Turing robot platform. If the real-time voice contains a preset second wake-up word, the control parameters are determined based on the wake-up word.
[0011] Furthermore, the real-time video stream is processed using a media pipeline algorithm to obtain key gesture points and their coordinate information.
[0012] Furthermore, the real-time video stream is captured by a camera mounted on a two-dimensional servo gimbal, wherein the two-dimensional servo gimbal is controlled by a palm tracking algorithm to achieve real-time tracking of the palm by the camera.
[0013] Secondly, the present invention provides a smart robot for home kitchens; A smart robot for home kitchens includes: The data acquisition module is used to acquire real-time audio or video streams; The main control module is used to execute the steps of the control method; The sub-control module is used to acquire control parameters and issue control commands based on the control parameters to control the smart home kitchen robot. The execution module is used to perform corresponding operations based on the control commands issued by the sub-control module.
[0014] Furthermore, the sub-control module includes a control unit and an intelligent robot platform. The control unit is used to generate control signals based on control parameters to control the operating status of the home kitchen intelligent robot, and the intelligent robot platform is used to generate user demand information based on the control parameters.
[0015] Thirdly, the present invention provides an electronic device; An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, complete the steps of the above-described home kitchen intelligent robot control method.
[0016] Fourthly, the present invention provides a computer-readable storage medium; A computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above-described home kitchen intelligent robot control method.
[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. The technical solution provided by this invention addresses the inconvenience faced by special groups such as the elderly and disabled in using intelligent devices by offering dual voice and gesture control, allowing users to choose the most comfortable way to interact with the device. Whether the user is a teenager or an elderly person, they can easily master the usage skills and quickly get started. This simple operation extends the practicality of the device to different user groups. Users who are not familiar with IoT technology or have some difficulty using intelligent devices can enjoy an efficient, intelligent, and convenient cooking experience, meeting the needs of different users.
[0018] 2. The technical solution provided by this invention aims to meet users' personalized needs and avoid the phenomenon of recipes on the market being monotonous and unable to be customized. By connecting to a backend intelligent cloud platform, it is equipped with powerful personalization capabilities to identify the preferences of different users and optimize recipes based on their tastes, eating habits, and other factors, bringing users a unique and personalized cooking experience that truly meets their needs.
[0019] 3. The technical solution provided by this invention integrates multiple devices and intelligent control modules to form a coordinated whole, thereby achieving full intelligence; it is connected to the Internet and has intelligent networking capabilities, enabling users to achieve a truly intelligent experience through remote operation and control of the devices.
[0020] 4. The technical solution provided by this invention uses high-performance, low-cost materials and technologies to reduce costs. Attached Figure Description
[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0022] Figure 1 An example diagram of dynamic gestures provided in an embodiment of the present invention; Figure 2 A schematic diagram of the dynamic gesture recognition process provided in an embodiment of the present invention; Figure 3 This is an example diagram of gesture key points obtained based on the MediaPipe algorithm provided in an embodiment of the present invention; Figure 4 A flowchart illustrating the four-level cascaded processing algorithm provided in an embodiment of the present invention; Figure 5 A schematic diagram of the speech recognition process provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the intelligent cloud platform provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the system architecture of a smart kitchen robot provided in an embodiment of the present invention. Detailed Implementation
[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0025] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0026] Example 1 The control and usage methods of existing intelligent kitchen robots are not suitable for the elderly and teenagers who are not familiar with operating intelligent devices, and most of the recipes on the internet are generic and cannot be customized to meet people's special needs. Therefore, this invention provides a method for controlling a home kitchen intelligent robot, which adopts dual control of voice and gestures and connects to an intelligent cloud platform to customize personalized recipes according to user needs.
[0027] Next, combined Figures 1-7 This embodiment discloses a method for controlling a smart home kitchen robot, which includes the following steps: S1. Acquire and recognize real-time voice or video streams, determine control parameters, and control the smart home kitchen robot based on the control parameters.
[0028] Furthermore, if the user controls the smart home kitchen robot through dynamic gestures, the system acquires and identifies real-time video streams, determines control parameters, and then controls the smart home kitchen robot based on these parameters.
[0029] The specific steps include: Step 1: Acquire real-time video stream and process the real-time video stream using a gesture recognition algorithm to obtain key gesture points and their coordinate information.
[0030] Specifically, the OpenCV library (a cross-platform computer vision library) is used to obtain real-time video streams from cameras, and the MediaPipe algorithm is used to process the real-time video streams to obtain key gesture points and their coordinate information.
[0031] Furthermore, the real-time video stream is captured by a camera mounted on a two-dimensional servo gimbal. The two-dimensional servo gimbal is controlled by a palm tracking algorithm to achieve real-time tracking of the palm by the camera.
[0032] Because home kitchen smart robots have high real-time requirements, tracking the human hand before processing dynamic gestures can greatly improve the system's real-time performance.
[0033] Therefore, in this embodiment, a palm tracking algorithm based on MediaPipe is innovatively designed based on the gesture key points obtained by the MediaPipe algorithm. The camera is fixed to a two-dimensional gimbal, providing real-time tracking of the palm in two dimensions. An incremental PID algorithm is used to derive the control value of the two-dimensional servo gimbal based on the difference between the coordinates of the palm's center point and the screen's center point, thereby achieving real-time tracking of the palm by the camera.
[0034] For example, the specific process is as follows: Step 1: Calculate the error value between the key points of the hand and the center position of the screen in the current frame.
[0035] Derived from the MediaPipe algorithm Figure 3 The coordinates of keypoint 9 are taken as the center point coordinates of the palm in the current frame. Then, the error value between the palm keypoint coordinates and the screen center position coordinates in the current frame is calculated, as follows:
[0036]
[0037] Where 160 and 120 are the x and y values of the center position point of the window, pid_x and pid_y are the center point coordinates of the palm in the current frame, and pid_thisError_x and pid_thisError_y are the error values between the key points of the palm in the current frame and the center position coordinates of the screen.
[0038] Step 2: Calculate the PWM signal adjustment values of the two-dimensional gimbal servo in the horizontal and vertical directions of the current frame.
[0039] Based on the error value obtained in step one, the incremental PID control algorithm is used to calculate the value of the servo motor's PWM (Pulse Width Modulation) signal. To reduce the system's computational load and improve the tracking speed, this incremental PID control algorithm only uses the proportional and derivative terms. Iterative error processing between the palm center point and the screen center point is also performed, adjusting the output PWM signal based on the difference between the current error and the previous error for the next processing iteration. Finally, to adapt to this system and keep the output range within a suitable range, the servo motor's PWM signal value is divided by 100 to obtain the servo motor's PWM signal adjustment value. The specific calculation process is as follows:
[0040] Where pwm_x and pwm_y are the PWM signal values that the two-dimensional gimbal servo needs to achieve in the horizontal and vertical directions, pid_lastError_x and pid_lastError_y are the error values between the previous palm center point and the screen center point coordinates, and their initial values are 0; pid_XP and pid_YP are the PWM signal adjustment values of the two-dimensional gimbal servo in the horizontal and vertical directions.
[0041] Step 3: Calculate the final PID control output values of the two-dimensional gimbal servo in the horizontal and vertical directions of the current frame.
[0042] After obtaining the PWM signal adjustment values for the 2D gimbal servo in the horizontal and vertical directions, these values are accumulated into the final PWM output signal value. This ensures that the control signal is updated at every time step, allowing the final output to reflect the previous control effects. By accumulating the control signal, the system makes incremental adjustments to gradually move the 2D gimbal servo to the desired position. This eliminates long-term deviations and achieves smooth tracking of the 2D gimbal servo. The specific calculation process is as follows:
[0043]
[0044] Wherein, pid_X_P and pid_Y_P are the final PID control output values of the two-dimensional gimbal servo in the horizontal and vertical directions of the current frame, and their initial values are 0. Step 2: Process the key points of the gesture and their coordinate information through a four-level cascaded processing algorithm to obtain the gesture recognition result.
[0045] Specifically, the detailed process for step 2 is as follows: Step 201: Based on the key points of the gesture and their coordinate information, obtain the number of protruding key points of the fingertips; based on the number of protruding points, determine the number of fingers that extend.
[0046] For example, in combination Figure 3 , Figure 4 Calculate the number of protruding key points of the fingers (i.e.) Figure 3 (Key points 4, 8, 12, 16, 20) First, calculate the number of fingers that are extended to obtain the first-level classification result.
[0047] For example: when the number of prominent key points at the fingertips is 0, it is initially judged as gesture 0 or 7; when the number of prominent key points at the fingertips is 1, it is judged as gesture 1 or 9; when the number of prominent key points at the fingertips is 2, it is judged as gesture 2, 6, 8, or 7; when the number of prominent key points at the fingertips is 3, it is judged as gesture 3; when the number of prominent key points at the fingertips is 4, it is judged as gesture 4 or 7; and when the number of prominent key points at the fingertips is 5, it is judged as gesture 5 or 7. The first concatenated result is then transmitted to the second concatenation.
[0048] Step 202: Based on the number of extended fingers, key finger points and their coordinates, obtain the angle information between the key finger points, and determine the degree of bending of a single finger joint based on the angle information.
[0049] For example, the degree of bending between different key points of a single finger is calculated to determine whether a single finger is bent. The first cascade result is then verified to obtain the second cascade result.
[0050] For example: when the result of the first cascade is 1 or 9, therefore determining whether the index finger is straight involves calculating the key points on the index finger (i.e., Figure 3 By differentiating the angles between key points 6, 7, and 8, we can initially distinguish between the number gestures 1 and 9. When the angle between key points 6, 7, and 8 is 0°, it is the number gesture 1; when the angle between key points 6, 7, and 8 is an obtuse angle, it is the number gesture 9. When the first concatenation result is 0 or 7, gesture 7 and gesture 0 are easily confused when the gesture is facing the camera. Therefore, we can calculate the key points on the thumb (i.e., Figure 3 The angles between key points 2, 3, and 4 are distinguished. If the angle between key points 2, 3, and 4 is 0°, it is the number gesture 7; if the angle between key points 2, 3, and 4 is an obtuse angle, it is the number gesture 0. The angles of the remaining fingers are calculated separately to obtain the number of outstretched fingers, which is used to verify the first cascade result and obtain the second cascade result.
[0051] Step 203: Based on the degree of bending of a single finger joint, obtain the angle between different fingers to determine the preliminary gesture recognition result.
[0052] For example, the angles between different fingers and the same joint are calculated to verify the second cascade result. For instance, if the second cascade result shows that two fingers are straight, it can be verified by calculation... Figure 3 The angles of key points 4, 8, and 20 are used to distinguish the number gestures 2, 6, and 8. The angles of key points 2, 0, and 20 are used to distinguish the number gestures 2 and 7. The second concatenation result shows that when four fingers are extended, the angles of key points 2, 0, and 20 can be used to distinguish the number gestures 4 and 7. Similarly, when five fingers are extended, the angles of key points 2, 0, and 20 can be used to distinguish the number gestures 5 and 7. If the second concatenation result shows that 0, 1, or 3 fingers are extended, angle calculation is unnecessary, and the value of the second concatenation result is directly output. After this stage of processing, the number gestures 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9 can be initially distinguished, resulting in the third concatenation result.
[0053] Step 204: Calculate the ratio of distances between fingertip key points based on their coordinates; obtain the final gesture recognition result based on the distance ratio and the preliminary gesture recognition result.
[0054] For example, for gestures with low recognition success rates, such as gestures of numbers 1 and 9, gestures of numbers 4, 5 and 7, this level of processing greatly increases the success rate, thus yielding the final number gesture.
[0055] For example, for gesture numbers 4 or 7, the ratio of the distance between key points 4 to 8 and key points 4 to 20 is calculated. If the ratio is greater than or equal to 0.8 and less than or equal to 1.5, it is gesture number 4; otherwise, it is gesture number 7. For gesture numbers 5 or 7, the same ratio is calculated. If the ratio is greater than or equal to 0.45 and less than or equal to 0.95, it is gesture number 5; otherwise, it is gesture number 7. For gesture numbers 1 or 9, the ratio of the distance between key points 0 to 8 and key points 0 to 5 is calculated. If the ratio is greater than or equal to 1.5, it is gesture number 1; otherwise, it is gesture number 9. Gesture recognition methods based on deep learning frameworks such as TensorFlow and PyTorch require a large amount of computing resources. Gesture recognition consuming too many computing resources will affect other functions, reduce the processing speed of home kitchen smart robots, and require a large amount of training data. Furthermore, the recognition success rate suffers to some extent when applied in different background environments; additionally, the recognition success rate for dynamic gestures is lower because dynamic gesture models are more complex, and complex models are prone to overfitting, resulting in poor performance in applications. The gesture recognition method based on OpenCV's convex hull is inaccurate, and its performance is even worse for dynamic gestures.
[0056] Based on the above analysis, this embodiment ultimately determined the dynamic gesture recognition scheme: an OpenCV + MediaPipe dynamic gesture recognition scheme. OpenCV is used to acquire real-time video streams from the camera, MediaPipe is used to acquire gesture key points, a hand tracking algorithm is added, and an innovative four-level cascaded processing algorithm for gesture key points is designed. After multiple tests, this scheme demonstrates fast processing speed, low computational resource consumption, and high accuracy in real-time dynamic gesture recognition. Ambient lighting and background factors have minimal impact on gesture key point recognition. After the cascaded processing algorithm, the recognition success rate is significantly improved in cases of overlapping and occlusion of user gestures. The accuracy of subsequent dynamic gesture recognition is also greatly improved.
[0057] Step 3: Compare the gesture recognition result with the preset dynamic gesture meaning to determine the control parameters, and transmit the control parameters to the sub-control module so that the sub-control module can generate control commands based on the control parameters and drive the execution module to perform the corresponding operation.
[0058] The term "dynamic gestures" is user-defined and includes functions such as dynamic gestures for page turning and controlling peripherals. Figure 1 As shown, the change from gesture number 0 to gesture number 8 is a dynamic gesture for opening the trash can.
[0059] Furthermore, in some embodiments, when a user controls a smart home kitchen robot via voice, real-time voice data is acquired and recognized to determine control parameters, which are then used to control the robot. Specific steps include: Step 1: Acquire and recognize real-time speech, which is collected and recognized by the speech receiving and broadcasting module.
[0060] In this embodiment, the voice receiving and broadcasting module is the SU-03T offline voice recognition module. This module processes front-end signals. Since the smart kitchen robot is used in the kitchen, there is often fixed-frequency noise, such as noise from range hoods and fans. This module has noise reduction methods for these fixed-frequency noises, providing effective noise shielding. The module also has a self-learning function, achieving high recognition accuracy for dialects and colloquial speech after self-learning, making the device's voice interaction more intelligent.
[0061] Step 2: Compare the recognized voice information with the preset wake-up word library to determine the control mode; determine the control parameters based on the control mode and real-time voice.
[0062] The wake-up word library includes functions such as starting a chat, helping me customize recipes, controlling peripherals, browsing recipes, and setting timers.
[0063] Specifically, if the voice message includes "start chatting," control parameters are generated to establish interaction with the intelligent chatbot platform; if the voice message includes "help me customize a recipe," control parameters are generated to establish interaction with the Turing robot platform. After establishing interaction with the intelligent chatbot platform or the Turing robot platform, the user speaks their specific needs, connects to the Baidu speech recognition interface, converts the voice message into text, and then sends the text message to the corresponding interaction platform.
[0064] Among them, the intelligent chatbot platform is Baidu UNIT intelligent dialogue platform.
[0065] For example, by calling the Turing Robot's API interface, you can achieve the function of interacting with the robot through dialogue; input the Key and Secret parts of the intelligent chatbot platform's secret key into the token() function to generate a URL, specify the location and access method of the resource to be accessed, so as to connect with the intelligent chatbot platform on the network.
[0066] When the voice message includes requests for customized recipes and specific recipe customization requirements, construct an HTTP POST request, passing the user's input information as a parameter to the Turing Robot API. The request URL is http: / / openapi.tuling123.com / openapi / api / v2. Add the necessary request headers to the HTTP request, including the API key and the request's Content-Type. The API key is used for authentication, and the Content-Type specifies the data format of the request, typically application / json.
[0067] The API processes the user's input and returns a JSON response containing the bot's reply. Parse the JSON response to extract the bot's reply. Typically, the reply is contained in specific fields of the JSON response; refer to the API documentation for details. Display the bot's reply to the user, either through a command-line interface, a webpage, or another application.
[0068] Furthermore, the robot's response is output to a text document, and a screenshot is taken using the `screenshot = pyautogui.screenshot()` function from the pyautogui library. Finally, the captured image is placed in the appropriate folder. When a recipe display command is received, the corresponding recipe image is opened for better viewing during cooking.
[0069] When the voice message includes the start of a chat, and the user further requests information such as news, jokes, or weather, the original voice is recorded and saved as a .wav file. The recording is then saved to the file myvoices.wav by calling the stop(self) method in the Recorder() class, and then uploaded to the Baidu Cloud platform via an HTTP POST request. Once the Baidu Cloud platform receives the audio file, it first preprocesses the audio, including noise removal, normalizing the audio volume, and segmenting the audio into different speech segments. Next, features are extracted from the preprocessed audio. After the model training is complete, a decoding algorithm can be used to convert the input feature sequence into the most probable text sequence.
[0070] The recognized text is then transmitted to the UNIT bot for dialogue using the `baidu_unit(text_words)` function. After obtaining a token (using the `requests.post()` method to send an HTTP POST request and return a Response object), the `json()` method is used to parse the JSON data contained in the response object into a Python object, and the parsing result is assigned to the `result` variable. However, the value of the obtained `result` variable is too redundant, so it is necessary to use the function `result = result['result']['responses'][0]['actions']` to obtain the value corresponding to the 'actions' field in the server response and assign it to the `result` variable. Finally, the obtained response is printed out. Simultaneously, corresponding skills are added to the bot on the Baidu Cloud platform to give the chatbot the relevant functionality.
[0071] Furthermore, in some embodiments, the speech synthesis API of Baidu Cloud Platform is invoked to broadcast the information replied by the robot.
[0072] Furthermore, in some embodiments, if the voice information is for controlling a peripheral device, such as opening a trash can, then control parameters are determined based on the voice and transmitted to the sub-control module, so that the sub-control module generates control commands based on the control parameters and drives the execution module to perform the corresponding operation.
[0073] Example 2 Combination Figure 7 This embodiment discloses a smart home kitchen robot, including a data acquisition module, a main control module, a sub-control module, and an execution module. The data acquisition module is used to acquire real-time voice or real-time video streams. The main control module is used to execute the steps of the control method described in Embodiment 1. The sub-control module is used to acquire control parameters and issue control commands to control the smart home kitchen robot according to the control parameters. The execution module is used to execute corresponding operations according to the control commands issued by the sub-control module.
[0074] Furthermore, the sub-control module includes a control unit and an intelligent robot platform. The control unit generates control signals based on control parameters to control the operation of the home kitchen intelligent robot, while the intelligent robot platform generates user demand information based on the control parameters. The data acquisition module includes a voice receiving and broadcasting module and a camera. The voice receiving and broadcasting module collects voice data, and the camera collects video data.
[0075] In this embodiment, the control unit is a Raspberry Pi 4B board, the sub-control module is an STM32 control board, and the execution module includes devices such as smart trash cans and smart cabinets; the smart robot platform is a smart chatbot platform or a Turing robot platform.
[0076] Furthermore, in some embodiments, a gas monitoring module and a buzzer are provided. The gas monitoring module is communicatively connected to a sub-control module, and the sub-control module is communicatively connected to the buzzer. Real-time monitoring is performed on easily leakable kitchen gases such as natural gas and carbon monoxide. If the gas content exceeds the standard, the buzzer will sound an alarm.
[0077] In this embodiment, the gas monitoring modules are MQ-4 and MQ-7 modules.
[0078] Example 3 Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When the computer instructions are executed by the processor, they complete the steps of the above-described home kitchen intelligent robot control method.
[0079] Example 4 Embodiment 4 of the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above-described home kitchen intelligent robot control method.
[0080] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0083] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for controlling a smart robot in a home kitchen, characterized in that, include: Acquire and recognize real-time voice or video streams, determine control parameters, and control a smart home kitchen robot based on these parameters. Acquiring and identifying real-time video streams includes: Acquire real-time video stream, process the real-time video stream, and obtain key gesture points and their coordinate information; A four-level cascaded processing algorithm is used to process the key points of the gesture and their coordinate information to obtain the gesture recognition result; The gesture recognition results are compared with the preset meanings of dynamic gestures to determine the control parameters; The processing of gesture key points and their coordinate information using a four-level cascaded processing algorithm includes: Based on the key points of the gesture and their coordinate information, obtain the number of protruding key points of the fingertips; based on the number of protruding points, determine the number of fingers that extend. Based on the number of extended fingers, key finger points and their coordinates, obtain the angle information between key finger points, and determine the degree of bending of a single finger joint based on the angle information; Based on the degree of bending of individual finger joints, the angles between different fingers are obtained to determine the preliminary gesture recognition results; Based on the key points of the fingertips and their coordinate information, calculate the ratio of the distances between the key points of the fingertips; based on the distance ratio and the preliminary gesture recognition results, obtain the final gesture recognition result.
2. The home kitchen intelligent robot control method as described in claim 1, characterized in that, The specific steps for acquiring and recognizing real-time speech are: Acquire real-time voice and compare its features with a preset wake-up word library to determine the control mode; The control parameters are determined based on the control mode and real-time voice prompts.
3. The home kitchen intelligent robot control method as described in claim 2, characterized in that, The process of acquiring real-time voice and comparing its features with a preset wake-up word library to determine the control mode includes: If the real-time voice contains a preset first wake-up word, control parameters are generated to establish interaction with the intelligent chatbot platform or the Turing robot platform. If the real-time voice contains a preset second wake-up word, the control parameters are determined based on the wake-up word.
4. The home kitchen intelligent robot control method as described in claim 1, characterized in that, The real-time video stream is processed using a media pipeline algorithm to obtain key gesture points and their coordinate information.
5. The home kitchen intelligent robot control method as described in claim 1, characterized in that, The real-time video stream is captured by a camera mounted on a two-dimensional servo gimbal. The two-dimensional servo gimbal is controlled by a palm tracking algorithm to achieve real-time tracking of the palm by the camera.
6. A smart robot for home kitchens, characterized in that, include: The data acquisition module is used to acquire real-time audio or video streams; The main control module is used to execute the steps of the home kitchen intelligent robot control method as described in any one of claims 1-5; The sub-control module is used to acquire control parameters and issue control commands based on the control parameters to control the smart home kitchen robot. The execution module is used to perform corresponding operations based on the control commands issued by the sub-control module.
7. The intelligent kitchen robot as described in claim 6, characterized in that, The sub-control module includes a control unit and an intelligent robot platform. The control unit is used to generate control signals based on control parameters to control the operating status of the home kitchen intelligent robot, and the intelligent robot platform is used to generate user demand information based on the control parameters.
8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-5.
9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-5.
Citation Information
Patent Citations
A gesture interaction system based on double finger angle domain characteristics and a working method thereof
CN109597485A
Intelligent recognition method and intelligent interaction method for traditional culture gestures and related devices
CN110443167A
