Remote cooking method and device based on mixed reality and terminal equipment

Abstract

The application discloses a mixed reality-based remote cooking method and device and a terminal device. The method comprises the following steps: a mixed reality device receives an instruction for starting remote cooking; in response to the instruction for starting remote cooking, an option of multiple remote control modes is provided, the multiple remote control modes respectively corresponding to multiple working modes executable by a remotely set mechanical cooking device; the mixed reality device receives a gesture instruction of an operator and determines that the gesture instruction corresponds to a first remote control mode, so as to control the mechanical cooking device to enter a first working mode corresponding to the first remote control mode, wherein the mechanical cooking device is bound to a skeletal joint of the remote operator to remotely reproduce cooking actions of the remote operator; and the mixed reality device acquires current cooking actions of the operator to make the mechanical cooking device reproduce the current cooking actions. According to the embodiment of the application, remote cooking can be conveniently realized.

Description

Technical Field

[0001] This application relates to the field of mixed reality technology and the field of cooking automation technology, specifically to a remote cooking method, device, terminal equipment, computer-readable storage medium, and computer program product based on mixed reality. Background Technology

[0002] Mixed Reality (MR) technology is a further development of Virtual Reality (VR) technology. This technology enhances the realism of the operator's experience by introducing real-world scene information into the virtual environment and establishing an interactive feedback loop between the virtual world, the real world, and the operator.

[0003] On the other hand, current cooking operations can be completed entirely by hand or processed using a food processor. However, food processors can only perform simple operations, such as making pasta or stir-frying simple dishes. As for manual operations, the time spent cooking in modern life is gradually decreasing, but people still hope to eat a clean and delicious meal that they have prepared themselves. This contradiction remains unresolved.

[0004] Regarding the aforementioned problems in related technologies, no effective solutions have yet been found. The above is merely background information relevant to this application as known to the inventors and does not constitute an admission of prior art. Summary of the Invention

[0005] In view of this, embodiments of this application provide a mixed reality-based remote cooking method, terminal device, computer-readable storage medium, and computer program product to solve at least one technical problem. The mixed reality-based remote cooking method is applied to an operator terminal, where the operator terminal is a mixed reality device. The method includes: the mixed reality device receiving an instruction to start remote cooking; in response to the instruction to start remote cooking, the mixed reality device providing the operator with multiple remote control mode options on a screen, the multiple remote control modes corresponding to various working modes executable by a remotely set mechanical cooking device; the mixed reality device receiving a gesture instruction from the operator and determining that the gesture instruction corresponds to a first remote control mode; the mixed reality device sending information of the first remote control mode determined according to the gesture instruction to the mechanical cooking device to control the mechanical cooking device to enter a first working mode corresponding to the first remote control mode, wherein in the first working mode, the mechanical cooking device is bound to the skeletal joints of the remote operator to remotely reproduce the remote operator's cooking actions; the mixed reality device acquiring the operator's current cooking action so that the mechanical cooking device can reproduce the current cooking action.

[0006] Optionally, the plurality of remote control modes further includes a second remote control mode, which is used to enable the mechanical cooking device to enter a second working mode, wherein in the second working mode, the mechanical cooking device is able to perform a preset combination of cooking operations.

[0007] Optionally, if the mixed reality device determines that the operator's gesture command corresponds to the second remote control mode, the method further includes: the mixed reality device providing options for multiple cooking functions on a screen, the multiple cooking functions corresponding to multiple sets of cooking operation combinations that the mechanical cooking device can execute; the mixed reality device receiving the operator's gesture command and determining that the gesture command corresponds to a first cooking function; the mixed reality device sending information of the first cooking function determined according to the gesture command to the mechanical cooking device to control the mechanical cooking device to start executing a first set of cooking operation combinations corresponding to the first cooking function, wherein the first set of cooking operation combinations includes multiple specified cooking operations that the mechanical cooking device can execute, and the first cooking function is realized after the multiple specified cooking operations are completed.

[0008] Optionally, the cooking operation includes at least one of the following: opening and closing the door, turning the fire on and off, automatic path finding, identifying cooking materials, picking up and placing cooking materials, weighing ingredients, pre-treating ingredients, or cooking ingredients.

[0009] Optionally, the method further includes: the mechanical cooking device using image recognition to identify the type of ingredients or cooking tools, wherein the ingredients include food or seasonings.

[0010] Optionally, the mechanical cooking device includes a resistance detection device, and the method further includes: when the resistance detection unit detects that the resistance encountered by the mechanical arm of the mechanical cooking device exceeds a preset resistance threshold, the operation of the current mechanical arm is suspended.

[0011] Optionally, the method further includes: the mixed reality device receiving an image of the current cooking scene, the image of the current cooking scene being acquired and transmitted by a camera on the mechanical cooking device.

[0012] This application provides a mixed reality-based remote cooking device applied to an operator terminal, wherein the operator terminal is a mixed reality device, and the device includes:

[0013] The receiving module is used to receive commands to start remote cooking;

[0014] A module is provided for responding to the instruction to start remote cooking by providing the operator with options for multiple remote control modes on the screen, the multiple remote control modes corresponding to various working modes that can be executed by the remotely set mechanical cooking equipment;

[0015] The determination module is used to receive the operator's gesture commands and determine that the gesture commands correspond to the first remote control mode;

[0016] The sending module is used to send information of the first remote control mode determined according to the gesture command to the mechanical cooking device, so as to control the mechanical cooking device to enter the first working mode corresponding to the first remote control mode, wherein in the first working mode, the mechanical cooking device is bound to the skeletal joints of the remote operator so as to be able to remotely reproduce the cooking actions of the remote operator.

[0017] The acquisition module is used to acquire the operator's current cooking action so that the mechanical cooking equipment can reproduce the current cooking action.

[0018] This application provides a terminal device, which includes a processor and a memory storing computer program instructions. When the processor executes the computer program instructions, it implements the steps of the method described above.

[0019] This application provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the steps of the method described above.

[0020] This application provides a computer program product, which includes computer program instructions that, when executed by a processor, implement the steps of the method described above.

[0021] This application proposes a solution that enables remote control of mechanical cooking equipment using mixed reality devices, achieving remote cooking. It not only allows for complex cooking operations but also enables operation of the mechanical cooking equipment via mixed reality devices. Through the recognition of body joints and other parts, the movements of the mechanical cooking equipment and the mixed reality operator are essentially synchronized. This application can assist in cooking, allowing operators to cook dishes remotely during spare moments. Furthermore, this application enables precise remote control of the mechanical cooking equipment, allowing operators to perform cooking operations similar to those performed manually, achieving the same cooking results as if the person were actually in the kitchen, thus helping to meet people's personalized cooking needs. Attached Figure Description

[0022] To more clearly illustrate the implementation of the embodiments of this application, the accompanying drawings of the embodiments of this application will be briefly described below.

[0023] Figure 1 This is a schematic diagram of an AR system architecture based on a server and a terminal device, according to an embodiment of this application.

[0024] Figure 2This is a schematic diagram of a virtual-real image fusion method for AR navigation using a mobile app.

[0025] Figure 3 This is a flowchart of a remote cooking method based on mixed reality, according to an embodiment of this application.

[0026] Figure 4 This is a flowchart of a remote cooking method under the second remote control mode according to an embodiment of this application.

[0027] Figure 5 This is a structural block diagram of a remote cooking device based on mixed reality, according to an embodiment of this application.

[0028] Figure 6 This is a schematic diagram of the structure of a terminal device or server used to implement the remote cooking method based on mixed reality in the embodiments of this application.

[0029] Figure 7 This is a schematic diagram of the software structure of an exemplary terminal device according to an embodiment of this application. Detailed Implementation

[0030] The principles and spirit of this application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided to make the principles and spirit of this application clearer and more thorough, enabling those skilled in the art to better understand and implement the principles and spirit of this application. The exemplary embodiments provided herein are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments described herein without inventive effort are within the scope of protection of this application.

[0031] Those skilled in the art will understand that the embodiments of this application can be implemented as a system, apparatus, device, method, computer-readable storage medium, or computer program product. Therefore, this application can be specifically implemented in at least one of the following forms: entirely hardware, entirely software, or a combination of hardware and software. According to a specific embodiment of this application, this application claims protection for a remote cooking method, apparatus, terminal device, server, computer-readable storage medium, and computer program product based on mixed reality.

[0032] In this document, terms such as first, second, and third are used only to distinguish one entity (or operation) from another, and are not intended to require or imply any order or relationship between these entities (or operations).

[0033] The embodiments of this application can be applied to servers and terminal devices. Please refer to... Figure 1This diagram schematically illustrates an AR system architecture based on a server and terminal devices. The AR system architecture includes a server 10 and several terminal devices 20. In some examples, the terminal devices 20 are AR devices, which can be dedicated AR devices such as head-mounted displays (HMDs), smart gloves, clothing, and other smart wearable electronic devices. In other examples, the terminal devices 20 can be general-purpose AR devices, such as mobile phones, laptops, tablets, virtual reality (VR) devices, in-vehicle devices, navigation devices, gaming devices, etc.

[0034] Taking AR helmets or AR glasses as an example, a head-mounted display, machine vision system, and mobile computer can be integrated into a wearable device. This device has a display resembling glasses and is worn on the user's head. It transmits augmented reality information to the display or projects it onto the user's eyes, enhancing the user's visual immersion. In some examples, AR devices also include cameras, which can be wide-angle, telephoto, or structured light cameras (also known as point cloud depth cameras, 3D structured light cameras, or depth cameras). Structured light cameras, based on 3D vision technology, can acquire the planar and depth information of objects. They project light with specific structural features onto the object using a near-infrared laser, then an infrared camera collects the reflected light, which is processed by a processor chip. The calculation principle involves calculating the object's position and depth information based on changes in the light signal caused by the object, presenting a 3D image. Typical terminal devices, such as mobile phones, display two-dimensional images and cannot show the depth of different locations within the image. Structured light cameras can capture and acquire 3D image information, obtaining not only color and other information at different locations but also depth information, which can be used for AR ranging. Of course, ordinary terminal devices can also acquire 2D images using optical cameras and combine this with deep learning algorithms to obtain depth information, ultimately displaying 3D images as well.

[0035] In some examples, terminal device 20 has AR-enabled software or an application (APP) installed. Server 10 can be a management server or application server for this software or APP. Server 10 can be a single server, a server cluster consisting of multiple servers, or a cloud server, etc. Terminal device 20 integrates modules with networking capabilities, such as Wireless-Fidelity (Wi-Fi) modules, Bluetooth modules, 2G / 3G / 4G / 5G communication modules, etc., to connect to server 10 via a network.

[0036] For example, an operator can log in to their account via an app installed on their mobile phone, or via software installed on AR glasses.

[0037] Taking an AR navigation app as an example, the app can possess capabilities such as high-precision map navigation, environmental understanding, and virtual-real fusion rendering. The app can report its current geographical location information to the server 10 through the terminal device 20, and the server 10 provides AR navigation services to the user based on the real-time geographical location information. For example, if the terminal device 20 is a mobile phone, in response to the user launching the app, the mobile phone can activate its camera to capture images of the real environment. Then, the system performs AR enhancement on the real environment images captured by the camera, integrating or overlaying rendered AR effects (such as navigation route markers, road names, merchant information, and advertising displays) into the real environment images, and displaying the virtual-real fusion image on the mobile phone screen.

[0038] Figure 2 The illustration schematically shows a virtual-real fusion image for AR navigation using a mobile app, where the AR navigation pointer arrows are superimposed on the real road surface and space in the image, and the electronic promotional materials of merchants float in the space in the form of parachutes carrying gift boxes at designated locations.

[0039] Embodiments of this application relate to terminal devices and / or servers. The principles and spirit of this application will be explained in detail below through several exemplary embodiments or representative implementations.

[0040] refer to Figure 3 This application proposes a remote cooking method based on mixed reality, applied to an operator's terminal, including the following steps:

[0041] S101, the mixed reality device receives a command to start remote cooking;

[0042] S102, in response to the command to start remote cooking, the mixed reality device provides the operator with multiple remote control mode options on the screen, and the multiple remote control modes correspond to the various working modes that can be performed by the remotely set mechanical cooking device;

[0043] S103, the mixed reality device receives the operator's gesture command and determines that the gesture command corresponds to the first remote control mode;

[0044] S104, the mixed reality device sends information of the first remote control mode determined according to the gesture command to the mechanical cooking device to control the mechanical cooking device to enter the first working mode corresponding to the first remote control mode. In the first working mode, the mechanical cooking device is bound to the skeletal joints of the remote operator to remotely reproduce the cooking actions of the remote operator.

[0045] S105, the mixed reality device acquires the operator's current cooking action so that the mechanical cooking device can reproduce the current cooking action.

[0046] This application proposes a solution that enables remote control of mechanical cooking equipment using mixed reality devices, allowing for cooking from a distance. The mechanical cooking equipment can be operated via mixed reality devices, using body joints and other means.

[0047] This application enables precise recognition, achieving near-synchronous movement between the mechanical cooking equipment and the operator's actions in a mixed reality device. It can also assist in cooking, allowing operators to remotely cook dishes during their leisure time. This application can also...

[0048] This enables remote and precise control of mechanical cooking equipment, allowing operators to perform cooking operations remotely similar to those performed manually, achieving the same cooking results as if the person were in the kitchen, thus enriching the variety of dishes.

[0049] According to one embodiment of this application, the plurality of remote control modes further includes a second remote control mode.

[0050] The second remote control mode is used to enable the mechanical cooking device to enter a second working mode, wherein in the second working mode, the mechanical cooking device is able to perform a preset combination of cooking operations.

[0051] This application also includes built-in preset cooking operation combinations, which allow for direct processing of ingredients according to pre-defined procedures. This eliminates the need for tedious manual operations, saving time and effort, when dealing with tasks that can be performed with fixed steps (such as chopping or peeling).

[0052] According to one embodiment of this application, if the mixed reality device determines that the operator's gesture command corresponds to the second remote control mode, the method further includes: the mixed reality device providing options for multiple cooking functions on a screen, the multiple cooking functions corresponding to multiple sets of cooking operation combinations that can be performed by the mechanical cooking device; the mixed reality device receiving the operator's gesture command and determining that the gesture command corresponds to a first cooking function; the mixed reality device then...

[0053] The information of the first cooking function determined by the control command is sent to the mechanical cooking device to control the mechanical cooking device to start executing a first set of cooking operation combinations corresponding to the first cooking function. The first set of cooking operation combinations includes multiple specified cooking operations that the mechanical cooking device can execute. After the multiple specified cooking operations are completed, the first cooking function is realized.

[0054] This application can realize complex cooking combinations, and achieve different cooking processes through different combinations.

[0055] The cooking combinations in this application can be edited, and different cooking operations can be combined arbitrarily to achieve the diversity of 5 dishes.

[0056] According to one embodiment of this application, the cooking operation includes at least one of the following: opening and closing the door, turning the ignition on and off, automatic pathfinding, identifying cooking materials, taking and placing cooking materials, weighing ingredients, pre-processing ingredients, or cooking ingredients.

[0057] This application includes various cooking operations, enabling the presentation of complex dishes. Furthermore, this application goes beyond simply processing ingredients; as it involves remotely controlled mechanical cooking equipment, it needs to perform operations similar to those of a human. Therefore, in addition to basic cooking, it requires functions that traditional cooking equipment cannot perform, such as retrieving ingredients from the refrigerator. Thus, this application can handle more complex cooking operations and offers greater operability.

[0058] According to one embodiment of this application, the method further includes: the mechanical cooking device using image recognition to identify the type of ingredients or cooking tools, wherein the ingredients include food or seasonings.

[0059] This application can automatically identify and locate the corresponding cooking tools and ingredients, eliminating the need for the operator to search for them one by one on the screen, making the operation more convenient.

[0060] According to one embodiment of this application, the mechanical cooking device includes a resistance detection device, and the method further includes: when the resistance detection unit detects that the resistance encountered by the mechanical arm of the mechanical cooking device exceeds a preset resistance threshold, the operation of the current mechanical arm is suspended.

[0061] In practice, since the operator doesn't handle the physical food, situations inevitably arise where the operation doesn't match the environment of the mechanical cooking equipment. For example, when cutting food, if the operator's cutting motion is too large and not controlled, directly replicating it will cause the mechanical cooking equipment to make excessive cutting motions. This could damage not only the knife and cutting board but also the mechanical cooking equipment itself, posing a certain danger. This application can mitigate this danger by stopping the current action when the mechanical cooking equipment encounters sufficient resistance, until the resistance falls below a resistance threshold.

[0062] According to one embodiment of this application, the method further includes: the mixed reality device receiving an image of the current cooking scene, the image of the current cooking scene being acquired and transmitted by a camera on the mechanical cooking device.

[0063] This application can send the current cooking process to a mixed reality device, allowing the operator to monitor the cooking progress. The operator can supervise whether the current operation meets the requirements and observe the degree of completion of the dish.

[0064] To more clearly illustrate the advantages that can be obtained from the embodiments of this application, the processing procedure of the embodiments of this application will be described in detail below based on specific examples.

[0065] As an example, when an operator uses a mixed reality device to perform a cooking operation, they should first ensure that the mixed reality device and the mechanical cooking device are connected to the network to ensure normal communication between them.

[0066] An operator remotely wakes up a mechanical cooking device using a mixed reality device (MRD). The operator then selects from multiple operating modes displayed on the MRD screen using gesture commands. Each of these remote control modes corresponds to a specific operating mode that the remotely configured mechanical cooking device can execute. For example, when the operator selects the first remote control mode, the mechanical cooking device enters that first operating mode. According to one embodiment of this application, the method includes at least two modes: a first remote control mode and a second remote control mode. According to another embodiment of this application, the MRD can recognize the operator's gesture commands and determine that the gesture commands correspond to one of the multiple remote control modes.

[0067] When the operator selects the first remote control mode, the mechanical cooking device enters the first working mode. According to one embodiment of this application, in the first working mode, the mechanical cooking device is bound to the skeletal joints of the remote operator to remotely reproduce the operator's cooking actions. A mixed reality device acquires the operator's current cooking action so that the mechanical cooking device can reproduce the current cooking action. According to one embodiment of this application, the mechanical cooking device has two robotic arms that can synchronously complete the operator's actions. According to one embodiment of this application, when the mechanical cooking device is in the first working mode, the joints of the robotic arms are bound to the joints of the operator's hand. The mixed reality device captures the motion trajectory data of the operator's arm and forearm joints, generates corresponding joint motion data, and sends it to the mechanical cooking device. Based on the motion trajectory data, the mechanical cooking device controls the movement of the corresponding robotic arm joints to replicate the operator's actions.

[0068] According to one embodiment of this application, the first remote control mode is a mode that synchronizes the operator's actions on the mechanical cooking equipment. In this mode, after the operator performs a certain action, the robotic arm of the mechanical cooking equipment will also perform a similar action. For example, when the operator wants to burn off any remaining animal hair on the meat skin (this action may not be included in the stored cooking operations), they can see the image captured by the mechanical cooking equipment through a mixed reality device, or generate a virtual object corresponding to the image captured by the mechanical cooking equipment through a mixed reality device. For example, the operator can see a virtual cutting board, a piece of meat, and a preheated pot through a virtual reality device. The operator can pick up the virtual piece of meat and place the skin side down the bottom of the virtual pot. At this time, the mechanical cooking equipment will also pick up the real piece of meat and place the skin side down the bottom of the real pot. The operator can pick up the meat during this process and observe the heating of the skin until the skin reaches the desired temperature.

[0069] According to one embodiment of this application, the multiple remote control modes further include a second remote control mode, which is used to enable the mechanical cooking device to enter a second working mode, wherein in the second working mode, the mechanical cooking device can execute a preset combination of cooking operations. According to one embodiment of this application, the mixed reality device provides multiple cooking function options on a screen, and the multiple cooking functions correspond to multiple sets of cooking operation combinations that the mechanical cooking device can execute. According to one embodiment of this application, the mixed reality device receives a gesture command from an operator and determines that the gesture command corresponds to a first cooking function. The mixed reality device sends information about the first cooking function determined according to the gesture command to the mechanical cooking device to control the mechanical cooking device to begin executing a first set of cooking operation combinations corresponding to the first cooking function. The first set of cooking operation combinations includes multiple specified cooking operations that the mechanical cooking device can execute; after the multiple specified cooking operations are completed, the first cooking function is achieved.

[0070] For mechanical cooking equipment, some cooking operations can be pre-stored, such as: opening and closing the door, turning the flame on and off, automatic path finding, identifying cooking materials, picking up and placing cooking materials, weighing ingredients, pre-processing ingredients, or cooking ingredients. According to one embodiment of this application, the mechanical cooking equipment executes a first set of cooking operation combinations in a first cooking function based on cooking control instructions, wherein the first set of cooking operation combinations includes at least one cooking operation.

[0071] For example, when preparing to process a large piece of meat, the operator can enter a second remote control mode and select the operation for the meat. This operation could be a cooking operation, such as pre-processing the ingredients, which involves cutting the large piece of meat into smaller pieces. Alternatively, it could be washing or peeling certain vegetables. This operation could also be a combination of first-group cooking operations, such as cutting the large piece of meat into pieces, blanching it, and then washing it. It could also be a first-group cooking function, such as making braised pork. The process of making braised pork may involve multiple combinations of first-group cooking operations. When the mechanical cooking equipment receives the corresponding instructions, it will process the large piece of meat according to the predetermined process until all cooking operations are completed.

[0072] During this process, the mechanical cooking equipment can identify the current state of the ingredients (shape, size, degree of cooking, etc.) and may determine whether the ingredients meet the requirements of the next step before proceeding. For example, when caramelizing sugar, it will determine whether the current caramel color meets the requirements to decide whether to continue caramelizing or proceed to the next step. According to one embodiment of this application, when a problem occurs in a certain step, such as a certain seasoning being missing, the mechanical cooking equipment will generate corresponding alarm information and feed it back to the mixed reality device.

[0073] This application utilizes mixed reality devices to remotely control mechanical cooking equipment, enabling remote cooking. It not only automates complex cooking operations but also allows operation of the mechanical cooking equipment via mixed reality devices. Through the recognition of body joints and other parts, the mechanical cooking equipment's movements are synchronized with those of the operator using mixed reality. This application can assist in cooking, allowing operators to remotely prepare dishes during their spare time. Furthermore, this application enables precise remote control of the mechanical cooking equipment, allowing operators to perform cooking operations similar to those performed manually, achieving a level of skill comparable to being physically present in the kitchen, thus enriching the variety of dishes that can be prepared.

[0074] Those skilled in the art will understand that the embodiments described herein are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily essential to the embodiments of this application. In the above embodiments, the descriptions of each embodiment have different focuses, and parts not described in detail in a certain embodiment can be referred to the relevant descriptions of other embodiments.

[0075] refer to Figure 5 This application also provides a remote cooking device 100 based on mixed reality, comprising:

[0076] The receiving module 110 is used to receive the command to start remote cooking;

[0077] A module 120 is provided to provide the operator with multiple remote control mode options on the screen in response to the instruction to start remote cooking. The multiple remote control modes correspond to the various working modes that can be executed by the remotely set mechanical cooking equipment.

[0078] The determination module 130 is used to receive the operator's gesture command and determine that the gesture command corresponds to the first remote control mode;

[0079] The sending module 140 is used to send information of the first remote control mode determined according to the gesture command to the mechanical cooking device, so as to control the mechanical cooking device to enter the first working mode corresponding to the first remote control mode, wherein in the first working mode, the mechanical cooking device is bound to the skeletal joints of the remote operator so as to be able to remotely reproduce the cooking actions of the remote operator.

[0080] The acquisition module 150 is used to acquire the operator's current cooking action so that the mechanical cooking device can reproduce the current cooking action.

[0081] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 60 includes a processor 61, a memory 62, and a communication bus for connecting the processor 61 and the memory 62. The memory 62 stores a computer program that can run on the processor 61. When the processor 61 runs the computer program, it can execute or implement the steps of the methods in the various embodiments of this application. The electronic device 60 also includes a communication interface for receiving and sending data. The electronic device 60 can be a server as described in the embodiments of this application, or it can be a cloud server. The electronic device 60 can also be a terminal device or an AR device as described in the embodiments of this application. Where appropriate, the electronic device can also be referred to as a computing device.

[0082] In some embodiments, processor 61 may be a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), modem processor, image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, neural-network processing unit (NPU), etc. Processor 61 may also be other general-purpose processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors may be microprocessors or any conventional processor. The neural network processor (NPU), by drawing inspiration from biological neural network structures, can rapidly process input information and continuously learn itself. The NPU electronic device 60 can realize applications such as intelligent cognition, including image recognition, face recognition, semantic recognition, speech recognition, and text understanding.

[0083] In some embodiments, memory 62 may be an internal storage unit of electronic device 60, such as a hard disk or memory of electronic device 60; memory 62 may also be an external storage device of electronic device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on electronic device 60. Memory 62 may include both internal storage units and external storage devices of electronic device 60. Memory 62 can be used to store operating system, application programs, bootloader, data, and other programs, such as program code of computer programs. Memory 62 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 62 is used to store program code executed by electronic device 60 and data transmitted. Memory 62 can also be used to temporarily store data that has been output or will be output.

[0084] Those skilled in the art will understand that Figure 6 This is merely an example of electronic device 60 and does not constitute a limitation on electronic device 60. Electronic device 60 may include more or fewer components than shown, or combine certain components, or include different components, such as input / output devices, network access devices, etc.

[0085] Figure 7 This is a schematic diagram of the software structure of a terminal device according to an embodiment of this application. Taking the Android operating system as an example, in some embodiments, the Android system is divided into four layers: the application layer, the application framework layer (FWK), the system layer, and the hardware abstraction layer. The layers communicate with each other through software interfaces.

[0086] First, the application layer can include multiple application packages, which can be various application apps such as calling, camera, video, navigation, weather, instant messaging, education, etc., or they can be application apps based on AR technology.

[0087] Second, the Application Framework Layer (FWK) provides application programming interfaces (APIs) and programming frameworks for applications within the application layer. The application framework layer can include predefined functions, such as functions for receiving events sent by the application framework layer.

[0088] The application framework layer may include a window manager, a resource manager, and a notification manager, among others.

[0089] The window manager manages the windowed applications. It can determine the screen size, the presence of a status bar, lock the screen, and capture screenshots. The content provider stores and retrieves data, making it accessible to applications. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, and phonebook entries.

[0090] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0091] The notification manager allows applications to display notifications in the status bar. These notifications can be used to convey informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0092] In addition, the application framework layer may include a view system, which includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build the application. The display interface can consist of one or more views; for example, the display interface of a text notification icon may include a view for displaying text and a view for displaying images.

[0093] Third, the system layer can include multiple functional modules, such as sensor service modules, physical state recognition modules, 3D graphics processing libraries (e.g., OpenGLES), and so on.

[0094] The sensor service module monitors sensor data uploaded by various sensors at the hardware layer to determine the physical state of the phone; the physical state recognition module analyzes and recognizes operator gestures, faces, etc.; and the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0095] In addition, the system layer may include a surface manager and a media library. The surface manager manages the display subsystem and provides 2D and 3D layer blending for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as still image files.

[0096] Finally, the hardware abstraction layer is the layer between hardware and software. The hardware abstraction layer can include display drivers, camera drivers, sensor drivers, etc., used to drive the relevant hardware in the hardware layer, such as displays, cameras, and sensors.

[0097] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed, implement the steps of the method designed in the above embodiments.

[0098] This application also provides a computer program product, including a computer program or instructions, which, when executed, implement the steps of the method designed in the above embodiments. For example, the computer program product may be a software installation package.

[0099] Those skilled in the art should understand that the functions of the methods, steps, or related modules / units described in the embodiments of this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product, or by a processor executing computer program instructions. The computer program product includes at least one computer program instruction, which can be composed of corresponding software modules. These software modules can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, portable hard disk, read-only optical disc (CD-ROM), or any other form of storage medium well known in the art. The computer program instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media, or semiconductor media (e.g., SSDs).

[0100] Regarding the various devices / products described in the above embodiments, the modules / units included can be software modules / units, hardware modules / units, or a combination of both. For example, for devices / products applied to or integrated into a chip, all of its modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the chip, while the remaining modules / units can be implemented using hardware methods such as circuits. Similarly, for devices / products applied to or integrated into a terminal, all of its modules / units can be implemented using hardware methods such as circuits, or at least some modules / units can be implemented using software programs running on a processor integrated within the terminal, while the remaining modules / units can be implemented using hardware methods such as circuits.

[0101] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A mixed reality-based remote cooking method, characterized by, The method is applied to an operator terminal, which is a mixed reality device, and comprises the following steps: The mixed reality device receives an instruction to start remote cooking; In response to the instruction to start remote cooking, the mixed reality device provides options of multiple remote control modes on a screen to an operator, the multiple remote control modes corresponding to multiple working modes executable by a remotely set mechanical cooking device respectively; The mixed reality device receives a gesture instruction of the operator and determines that the gesture instruction corresponds to a first remote control mode; The mixed reality device sends information of the first remote control mode determined according to the gesture instruction to the mechanical cooking device to control the mechanical cooking device to enter a first working mode corresponding to the first remote control mode, wherein in the first working mode, the mechanical cooking device is bound to a skeletal joint of a remote operator to remotely reproduce cooking actions of the remote operator; The mixed reality device acquires a current cooking action of the operator to make the mechanical cooking device reproduce the current cooking action; The multiple remote control modes further include a second remote control mode, and the second remote control mode is used to make the mechanical cooking device enter a second working mode, wherein in the second working mode, the mechanical cooking device can execute a preset cooking operation combination, the mechanical cooking device can identify a current state of a food material, and before executing a next operation, it is determined whether the current food material meets a requirement of the next operation, the current state of the food material including at least one of a shape, a size, and a cooking degree of the food material; If the mixed reality device determines that the gesture instruction of the operator corresponds to the second remote control mode, the method further comprises the following steps: The mixed reality device provides options of multiple cooking functions on a screen, the multiple cooking functions corresponding to multiple groups of cooking operation combinations executable by the mechanical cooking device respectively; The mixed reality device receives a gesture instruction of the operator and determines that the gesture instruction corresponds to a first cooking function; The mixed reality device sends information of the first cooking function determined according to the gesture instruction to the mechanical cooking device to control the mechanical cooking device to start executing a first group of cooking operation combinations corresponding to the first cooking function, wherein the first group of cooking operation combinations includes multiple specified cooking operations executable by the mechanical cooking device, and the multiple specified cooking operations are completed to realize the first cooking function.

2. The method of claim 1, wherein, The cooking operation includes at least one of opening and closing a door, opening and closing a fire, automatic route searching, cooking material identification, cooking material taking and placing, food material weighing, food material preprocessing, and food material cooking.

3. The method of claim 2, wherein, The method further comprises the following steps: The mechanical cooking device identifies a kind of food material or cooking tool through image recognition, wherein the food material includes food or seasoning.

4. The method of claim 1, wherein, The mechanical cooking device includes a resistance detection device, and the method further comprises the following step: when the resistance detection unit detects that resistance received by movement of a mechanical arm of the mechanical cooking device exceeds a preset resistance threshold, the current operation of the mechanical arm is paused.

5. The method of claim 1, wherein, The method further comprises the following steps: The mixed reality device receives a picture of a current cooking scene, which is obtained by a camera on the mechanical cooking device and transmitted.

6. A mixed reality based remote cooking device, characterized in that, The device is applied to an operator terminal, which is a mixed reality device, and comprises: a receiving module configured to receive an instruction for starting remote cooking; a providing module configured to, in response to the instruction for starting remote cooking, provide, on a screen, options of a plurality of remote control modes corresponding to a plurality of working modes executable by a remotely set mechanical cooking device; a determining module configured to receive a gesture instruction of the operator and determine that the gesture instruction corresponds to a first remote control mode; a sending module configured to send information of the first remote control mode determined according to the gesture instruction to the mechanical cooking device, so as to control the mechanical cooking device to enter a first working mode corresponding to the first remote control mode, wherein in the first working mode, the mechanical cooking device is bound to a skeletal joint of a remote operator to remotely reproduce cooking actions of the remote operator; an obtaining module configured to obtain a current cooking action of the operator, so as to make the mechanical cooking device reproduce the current cooking action; wherein the plurality of remote control modes further comprises a second remote control mode, and the second remote control mode is used to make the mechanical cooking device enter a second working mode, wherein in the second working mode, the mechanical cooking device can execute a preset cooking operation combination, the mechanical cooking device can identify a current state of a food material, and before executing a next operation, it is determined whether the current food material meets a requirement of the next operation, and the current state of the food material includes at least one of a shape, a size, and a cooking degree of the food material; if the mixed reality device determines that the gesture instruction of the operator corresponds to the second remote control mode, the mixed reality device provides, on a screen, options of a plurality of cooking functions corresponding to a plurality of cooking operation combinations executable by the mechanical cooking device; the mixed reality device receives a gesture instruction of the operator and determines that the gesture instruction corresponds to a first cooking function; the mixed reality device sends information of the first cooking function determined according to the gesture instruction to the mechanical cooking device, so as to control the mechanical cooking device to start executing a first cooking operation combination corresponding to the first cooking function, wherein the first cooking operation combination includes a plurality of specified cooking operations executable by the mechanical cooking device, and the first cooking function is realized after the plurality of specified cooking operations are completed.

7. A terminal device, characterized by comprising: A processor and a memory storing computer program instructions are included, and the processor executes the computer program instructions to implement the method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the method in any one of claims 1-5.

9. A computer program product, characterised in that, The computer program instructions are executed by a processor to implement the method in any one of claims 1-5.

Images