Robot control method and device, robot and storage medium

Abstract

The invention is suitable for the technical field of robot control, and provides a robot control method and device, a robot and a storage medium, the robot is provided with at least one layer of trays used for bearing tableware, the method comprises the steps that when the robot reaches a preset cleaning area, each layer of tray is shot, and a first to-be-detected image is obtained, updating the first to-be-detected image at a preset frequency; judging the bearing state of each layer of tray based on the latest first to-be-detected image to obtain a first judgment result; and controlling the robot according to the first judgment result. The working efficiency of the robot can be improved.

Description

Technical Field

[0001] The present application belongs to the field of robot control technology, and in particular relates to a robot control method, a robot device, a robot, and a computer-readable storage medium. Background Art

[0002] With the rapid development of the service robot industry, the application areas of service robots are constantly expanding. The catering industry is a popular application area for service robots. When used in the catering industry, service robots are typically used to perform tasks such as delivering meals or collecting tableware. When performing a tableware collection task, the service robot first moves to the corresponding table to collect the tableware. When a user, such as a customer or waiter, inputs a command such as "return to the kitchen" or "clear tableware", the service robot transports the collected tableware to a designated area such as the kitchen. The waiter then clears the tableware loaded by the service robot so that the service robot can proceed to the next tableware collection task.

[0003] At present, when emptying the tableware loaded by the service robot, the waiter or other users are usually required to manually click the emptying completion button or enter the corresponding command to indicate that the tableware has been emptied. This is highly dependent on manual labor, and if the user forgets to actively click the emptying completion button, it will affect the normal operation of the service robot. Summary of the Invention

[0004] The embodiments of the present application provide a robot control method, device, robot, and storage medium, which can improve the robot's work efficiency.

[0005] In a first aspect, an embodiment of the present application provides a method for controlling a robot, wherein the robot is provided with at least one tray for carrying tableware, the method comprising:

[0006] When the robot reaches the preset cleaning area, it takes a picture of each layer of the pallet to obtain a first image to be tested, wherein the first image to be tested is updated at a preset frequency;

[0007] Determine the load status of each layer of the pallet based on the latest first image to be tested, and obtain a first determination result;

[0008] The robot is controlled according to the first judgment result.

[0009] In a second aspect, an embodiment of the present application provides a control device for a robot, wherein the robot is provided with at least one tray for carrying tableware, and the device comprises:

[0010] a first image to be tested acquisition module, configured to photograph each layer of the pallet when the robot reaches a preset cleaning area to obtain a first image to be tested, wherein the first image to be tested is updated at a preset frequency;

[0011] A first judgment module is configured to judge the load status of each layer of the pallet based on the latest first image to be tested, and obtain a first judgment result;

[0012] A control module is used to control the robot according to the first judgment result.

[0013] In a third aspect, an embodiment of the present application provides a robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the robot control method of the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the robot control method in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a robot, enables the robot to execute any one of the above-mentioned robot control methods in the first aspect.

[0016] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0017] In an embodiment of the present application, when the robot reaches a preset cleaning area in order to empty the tableware carried by a tray, it will capture a first image to be tested of each layer of the trays. The robot then determines the loading status of each layer of the trays based on the obtained first image to be tested, and obtains a corresponding first judgment result. That is, the robot actively determines the status of the tray carrying the tableware based on the first image to be tested, thereby determining whether the tray has been emptied of the tableware, without requiring a user to manually input a command to passively determine whether the tray has been emptied. This reduces manual reliance and improves the robot's work efficiency. Furthermore, when acquiring the first image to be tested, the first image to be tested is updated at a preset frequency, and each judgment is made based on the latest first image to be tested, ensuring the real-time and accuracy of the first judgment result. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art.

[0019] Figure 1 This is a flow chart of a robot control method provided by one embodiment of the present application;

[0020] Figure 2This is a schematic structural diagram of a robot provided in an embodiment of the present application;

[0021] Figure 3 is a schematic structural diagram of another robot provided in an embodiment of the present application;

[0022] Figure 4 This is a flow chart of a robot control method provided in an embodiment of the present application;

[0023] Figure 5 This is a flow chart of a robot control method provided in an embodiment of the present application;

[0024] Figure 6 Schematic diagram of the structure of the control device of the robot provided in the embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of the structure of the robot provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0028] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized.

[0031] Example 1:

[0032] Figure 1 A schematic flow chart of a robot control method provided by an embodiment of the present invention is shown, and is described in detail as follows:

[0033] Step S101 : When the robot arrives at a preset cleaning area, it takes a picture of each layer of the pallet to obtain a first image to be tested, wherein the first image to be tested is updated at a preset frequency.

[0034] The robot is provided with a tray for carrying tableware so that the user can place the tableware to be recycled on the tray for recycling. In order to improve the carrying capacity of the robot, at least two layers of trays can be provided on the robot, and each layer can include a tray.

[0035] For example, Figure 2 As shown, three pallets can be set at different heights of the robot: pallet A, pallet B and pallet C, that is, three layers of pallets are set, with one pallet on each layer.

[0036] During the robot's operation, when the tray is fully loaded, it will perform an emptying task, that is, go to a preset cleaning area (such as a sink or a cleaning room) to empty the tableware that has been carried. When emptying the tableware carried by the robot's tray, the user is usually required to manually empty the tableware on the tray, and then input the corresponding instructions through buttons, etc., indicating that the tableware has been emptied and the emptying task is completed. In this process, manual dependence is high and the robot's waiting time is long. In particular, staff in the cleaning area often need to wash the tableware. At this time, it is inconvenient to click on the robot screen with both hands, and the operation is difficult. In addition, due to water stains, gloves, etc., it is difficult to issue instructions through the robot screen.

[0037] To reduce manual reliance, when the robot reaches a preset cleaning area, it can control a pre-set camera device (such as a camera installed on the robot or a surveillance camera installed at a designated location) to capture the tray, obtaining a first image to be tested that reflects the loading state of the tray on the robot. This image can then be used to proactively determine whether the tray has been emptied of tableware. The loading state refers to the state of the tray carrying tableware, such as a partially loaded state and a fully loaded state.

[0038] Since it takes a certain amount of time for the user to empty the tray of tableware, continuous detection of the tray requires a large amount of computing resources of the robot. Therefore, in order to promptly understand whether the emptying task is completed while reducing the occupied computing resources, after the robot reaches the preset cleaning area, it can obtain the first image to be tested at a preset frequency, obtaining the first image to be tested corresponding to different times. In other words, it is equivalent to updating the first image to be tested at a preset frequency, ensuring the real-time performance of the first image to be tested. Through this processing, the real-time performance of the tray load status update is improved, and the robot's work efficiency is improved.

[0039] For example, assuming the preset frequency is 5 seconds / time, the robot can obtain a video stream obtained by shooting the pallet, and then intercept video frames from the video stream based on a frequency of 5 seconds / time, and use the intercepted new video frames as the new first image to be tested to update the first image to be tested.

[0040] Optionally, when the robot moves to the preset cleaning area, the camera does not capture the pallet image, which reduces resource usage and allows the robot resources to be fully used for navigation and positioning, thereby improving the accuracy of the robot's movement and reducing positioning failures.

[0041] Optionally, when the first image to be tested is updated at a preset frequency, in order to reduce storage space usage and lower storage capacity requirements for the robot, the camera device may be controlled to shoot the pallet at a preset frequency to update the first image to be tested.

[0042] In some embodiments, the tray and tableware can be distinguished by specific features (such as color, shape, or texture). For example, the tray can be set to the same specific color, such as blue, yellow, or gray, to reduce the interference caused by the same color of the tray and the tableware in subsequent judgment.

[0043] Optionally, in order to accurately determine whether the emptying task is completed, the position and number of the camera devices can be set in combination with the position and number of the trays on the robot.

[0044] For example, Figure 2As shown, cameras 1, 2, and 3 can be set in combination with the positions of pallet A, pallet B, and pallet C. Through the above settings, the pallets on each layer can be photographed well, improving the accuracy of the first image to be measured.

[0045] In other embodiments, the tray provided on the robot may be a see-through tray made of see-through material. Figure 3 As shown, when setting up cameras, one camera can be placed between every two trays to capture images of adjacent trays, reducing the number of cameras required and conserving resources. Furthermore, trays with transparent center edges are used to load tableware, while the edges are made of opaque material to facilitate identification of the loading area. When using opaque tableware, it is possible to easily and accurately determine whether a tray is loaded with tableware, improving efficiency and accuracy.

[0046] In an embodiment of the present application, when the robot reaches a preset cleaning area to empty the tableware on the tray, it obtains a first image to be tested that can reflect the loading status of the tray, so that when it is subsequently determined that the tableware on the tray has been emptied, the robot can be promptly controlled to perform tableware recovery and other tasks without waiting for manual confirmation by the user, thereby improving the robot's work efficiency.

[0047] Step S102 : judging the loading status of each layer of the pallet based on the latest first image to be tested, and obtaining a first judgment result.

[0048] The first judgment result is used to indicate the loading status of each tray. The loading status refers to the status of the tray carrying tableware, such as a partially loaded state and a fully loaded state. The partially loaded state can also include an empty state (i.e., all the tableware is emptied) and a semi-empty state (i.e., part of the tableware is emptied).

[0049] Specifically, after obtaining the first image to be tested, since the first image to be tested will be updated at a preset frequency, in order to ensure the accuracy of the judgment, each time a judgment is made, the load status of each layer of the robot's pallet can be judged based on the latest first image to be tested to obtain the corresponding first judgment result.

[0050] It can be understood that when the robot includes multiple layers of pallets, the first image to be tested corresponding to each layer of pallets can be obtained respectively, and the load status of each layer of pallets can be judged based on the corresponding first image to be tested.

[0051] For example, the loading status of the tray can be judged based on parameters such as the height of the tableware carried on the tray or the area of ​​the tableware carried on the tray relative to the area of ​​the tray. For example, when the height of the tableware is greater than or equal to a preset first height threshold (such as 10 cm), the tray is judged to be in a fully loaded state; when the height of the tableware is less than the preset first height threshold and greater than or equal to a preset second height threshold (such as 3 cm), the tray is judged to be in a semi-empty state; when the height of the tableware is less than the second height threshold, the tray is judged to be in an empty state. It should be noted that the second height threshold is less than the first height threshold.

[0052] In this embodiment, the first image to be tested is captured after the robot reaches the preset cleaning area and waits for the trays to be emptied, providing high real-time performance. Based on this first image, the loading status of each tray layer can be determined in real time to determine whether the trays are empty, ensuring the real-time and accuracy of the first determination result.

[0053] Step S103: controlling the robot according to the first judgment result.

[0054] Specifically, after obtaining the first judgment result corresponding to the latest first image to be tested, the robot is controlled to perform the corresponding task according to the first judgment result. For example, the robot can be controlled accordingly by judging whether it meets the preset requirements according to the first judgment result.

[0055] For example, if the preset requirement is that each tray on each layer is empty, if the latest first judgment result indicates that each tray on each layer of the robot is empty, that is, the robot has met the preset requirement and the cleaning task is completed, then the robot can be controlled to leave the preset cleaning area to perform other tasks, such as standby tasks. This ensures that the robot has sufficient space to carry tableware when leaving the preset cleaning area.

[0056] If the first judgment result indicates that the robot does not meet the preset requirements, the robot is controlled to stay in place and continue to wait until the latest first judgment result indicates that the robot meets the preset requirements, and then the robot is controlled to leave the preset cleaning area.

[0057] In an embodiment of the present application, when the robot reaches a preset cleaning area, it will capture each layer of trays installed on the robot to obtain a corresponding first image to be tested. The obtained first image to be tested can reflect the load status of the tray. Therefore, the load status of the tray can be actively determined directly based on the first image to determine whether the tray load status meets the set requirements, so that the robot can be controlled accordingly in a timely manner to improve work efficiency. For example, it can determine whether the tableware on the tray has been cleared. When the first judgment result indicates that the tableware on the tray has been cleared, the robot can be controlled accordingly in a timely manner, without the need for the user to manually input commands to passively determine whether the tableware on the tray has been cleared. This reduces manual dependence and improves the robot's work efficiency. In addition, when acquiring the first image to be tested, the first image to be tested is updated at a preset frequency. Each judgment is based on the latest first image to be tested, which can ensure the real-time and accuracy of the first judgment result.

[0058] It should be noted that when a restaurant is equipped with multiple robots, the aforementioned robot control method can also be applied to the server. To obtain a first test image corresponding to a particular robot, the server can control the corresponding camera to capture each layer of the robot's trays. The server then uses the captured image as the first test image corresponding to the robot. The server then performs a judgment based on the first test image to obtain a first judgment result. Finally, the server controls the robot based on the obtained first judgment result. For example, the server can send corresponding instructions to the robot to be controlled to achieve accurate control of the desired robot.

[0059] In some embodiments, the above step S103 includes:

[0060] When the latest first judgment result indicates that each layer of the pallets is in an empty state, a standby task is executed, and the standby task is used to control the robot to go to a preset waiting area to wait for a call.

[0061] Specifically, in order to ensure the emptying effect and enable the trays on the robot to carry more tableware, the standby task will be executed when the latest first judgment result indicates that the trays on each layer are in an empty state; that is, after determining that the tableware on each layer of the trays is emptied, the emptying task is determined to be completed, and the robot is controlled to go to the preset waiting area (such as the kitchen door) to wait for the user's call.

[0062] In the embodiment of the present application, the emptying task is determined to be completed only when the tableware on all trays of the robot has been emptied, and the standby task is executed so that when the robot leaves the preset cleaning area, all trays are in an empty state, thereby ensuring the tableware carrying capacity of the robot.

[0063] In some embodiments, when the robot is located in a preset cleaning area, the method further includes:

[0064] When receiving the clearing completion instruction input by the user, the above-mentioned standby task is executed.

[0065] While the robot waits for the tray to be cleared, the user can proactively notify the robot to leave the preset cleaning area by inputting commands (such as clicking the robot's end button or using voice input). This allows for more flexible control of the robot based on manual commands if the user needs to make room in the preset cleaning area, if they want the robot to perform standby tasks during busy periods to respond to recycling, or if the user discovers that the robot's automatic judgment is incorrect.

[0066] Since the first image to be tested needs to be updated at a preset frequency in the embodiment of the present application, the user may actively notify the robot to leave the preset cleaning area while waiting for the acquisition of a new first image to be tested. Therefore, in order to improve the working efficiency of the robot, during the period when the robot acquires two first images to be tested, if a clearing completion instruction input by the user is received, it can also be determined that the clearing task is completed, and the standby task can be executed to control the robot to go to the preset waiting area to wait, so as to improve the working efficiency of the robot.

[0067] That is, when the robot is in the preset cleaning area and waiting for the tableware it carries to be emptied, if it receives an emptying completion instruction input by the user, it no longer needs to obtain the first image to be tested for judgment, but can directly execute the standby task and control the robot to go to the preset waiting area to wait for the user's call.

[0068] In an embodiment of the present application, while the robot is waiting for the tableware it carries to be emptied, if it receives an emptying completion instruction actively input by the user, it can directly execute the standby task without further judgment until it determines that all trays are in an empty state, so as to ensure the robot's work efficiency.

[0069] In some embodiments, the above method further comprises:

[0070] When the robot is located in the preset cleaning area and / or the preset waiting area, if a call instruction is received, the robot takes a picture of each layer of the pallet to obtain a second image to be measured.

[0071] The load status of each layer of the pallet is determined based on the second image to be tested to obtain a second determination result.

[0072] When the second judgment result indicates that any layer of the tray is not fully loaded, the recycling task corresponding to the call instruction is executed, and the recycling task is used to control the robot to go to the target recycling area to recycle the tableware.

[0073] The above-mentioned call instruction is used to indicate that a user (such as a customer) has a need to recycle tableware.

[0074] Specifically, in order to improve the robot's response efficiency and ensure the user's dining experience, when the robot is in the preset cleaning area, if it receives a call command, it will take a picture of each layer of the tray to obtain the second test image corresponding to the current moment, and judge whether to respond to the call command in real time based on the second test image.

[0075] Among them, when judging whether to respond to the call instruction in real time based on the second image to be tested, the load status of each layer of the tray can be judged based on the second image to be tested to obtain the corresponding second judgment result. Because the call instruction reflects the user's need to recycle tableware. In the restaurant scenario, timely response to customer needs is crucial to user experience. Therefore, in order for the robot to be able to respond to user needs in a timely manner, if the second judgment result indicates that the tray of any layer of the robot is in a non-full state (such as an empty state or a semi-empty state), that is, if it is determined that the tray of any layer has the ability to carry tableware, the recycling task corresponding to the call instruction can be directly executed, that is, the robot is controlled to go to the target recycling area corresponding to the received call instruction to recycle the tableware. If there are multiple robots with trays in a non-full state at the same time, the robot with the largest carrying capacity is controlled to respond to the call.

[0076] It is understood that if the second judgment result indicates that all trays on any layer are fully loaded, the recovery task corresponding to the call instruction cannot be executed. In this case, the recovery task corresponding to the call instruction can be set as a pending task. If the robot is in the preset waiting area at this time, an emptying task can be generated, and the robot can be controlled to move to the preset cleaning area so that it can quickly execute the pending task after clearing the plates.

[0077] When the second judgment result indicates that the pallets on any layer are fully loaded, in order to ensure response speed, the second image to be tested can be continuously acquired for judgment. When the latest second judgment result indicates that the pallets on any layer are not fully loaded, the recycling task is executed.

[0078] Alternatively, after the robot completes its current recycling task and has no pending tasks, it will typically execute a standby task, awaiting a user's call. During the robot's emptying task, the user may mistakenly issue an emptying completion command, or may wish the robot to leave the cleaning area, causing the robot to execute the standby task without clearing the dishes. To prevent the robot from directly executing a recycling task when all trays on each layer are fully loaded, when the robot is in a preset waiting area and receives a call command, it can also capture each tray layer, obtaining a second image to be tested corresponding to the current moment, and based on the second image to be tested, determine whether to respond to the call command in real time. That is, based on the second image to be tested, determine the load status of each tray layer. If it is determined that any tray layer of the robot is not fully loaded, the robot will also respond to the call command in real time and execute the recycling task corresponding to the call command.

[0079] like Figure 4 As shown, in order to ensure the response speed of the robot, improve the user experience, and avoid the situation where the robot has just rewound to the preset cleaning area and has not started cleaning and is still in a fully loaded state, a second test image can be obtained in both of the above cases, and it can be determined based on the second test image whether to respond to the received call command in real time.

[0080] In the embodiment of the present application, upon receiving a call command, the second image to be tested corresponding to the current moment is acquired in real time to determine the load status of each tray layer. If the second determination result indicates that any tray layer is not fully loaded, the robot responds to the call command in real time, controlling the robot to move to the corresponding target recycling area to recycle the tableware, thereby improving the robot's response speed and ensuring user experience. Furthermore, when the robot is located in a preset cleaning area or a preset waiting area and does not need to move, upon receiving a call command, it takes a picture of each tray layer and determines its status. This avoids responding to calls when the robot is fully loaded and avoids taking pictures and determining its status while the robot is moving, thereby preventing excessive memory usage from affecting the normal operation of other functions such as robot movement.

[0081] In some embodiments, after executing the recycling task corresponding to the above call instruction, the method further includes:

[0082] When it is determined that the recycling task is completed, each layer of the tray is photographed to obtain a third image to be tested.

[0083] The load status of each layer of the pallet is determined based on the third image to be tested to obtain a third determination result.

[0084] When the third judgment result indicates that each layer of the tray is fully loaded, an emptying task is performed, and the emptying task is used to control the robot to go to the preset cleaning area to empty the tableware.

[0085] In order to further improve the working efficiency of the robot, when determining that the currently executed recycling task is completed, each layer of the robot's pallet can also be photographed to obtain a third image to be tested that can reflect the loading status of the pallet.

[0086] like Figure 5 As shown, after judging the loading status of each layer of the tray based on the real-time third test image and obtaining the third judgment result, if the third judgment result indicates that each layer of the tray is in a fully loaded state, the emptying task is performed, that is, the robot is controlled to go to the preset cleaning area to empty the loaded tableware. At the same time, it is ensured that the robot can respond as soon as possible when receiving the call instruction and perform the corresponding recycling task, thereby improving the robot's work efficiency while ensuring the user experience.

[0087] If the third judgment result indicates that any layer of the pallet is not fully loaded, the standby task can be executed to control the robot to go to the preset waiting area and wait for the user's call.

[0088] During the execution of a recycling task, the current recycling task can be determined to be completed upon receiving a recycling completion instruction input by the user (e.g., the tableware has been placed). In other embodiments, to prevent the robot from being occupied for a long time when the user forgets to manually confirm, thereby affecting the execution of other tasks, the current recycling task can be determined to be completed when the execution time of the current recycling task exceeds a preset time threshold (e.g., 2 minutes).

[0089] In an embodiment of the present application, when judging that the recycling task is completed, each layer of the tray on the robot is also photographed, and the loading status of the tray is judged based on the third image to be tested, so as to promptly clear the tableware when necessary, and avoid executing the standby task or the new recycling task when new tableware cannot be loaded, thereby ensuring the user experience while improving the work efficiency of the robot.

[0090] In some embodiments, the call instruction is issued by a call bell provided on the dining table. Before executing the recycling task corresponding to the call instruction, the process further includes:

[0091] The target recycling area is determined based on the dining table corresponding to the call bell that issued the call instruction.

[0092] The corresponding recycling tasks are determined based on the target recycling area.

[0093] Specifically, in order to facilitate users to call the robot to recycle tableware, a call bell can be set at each dining table, and the user can issue a call instruction by pressing the call bell set at the dining table.

[0094] Since the call bells correspond to the dining tables one by one, after receiving the call instruction, the dining table of the user who needs to be recycled can be determined according to the call bell that issued the call instruction, and then the target recycling area can be determined according to the area where the dining table is located.

[0095] After determining the target recycling area, the corresponding recycling task can be determined based on the target recycling area. When executing the recycling task, the robot can be accurately controlled to go to the corresponding table to collect the tableware. By setting a call bell on the table to issue a call command, customers can easily call the robot according to their dining situation and needs, reducing the workload of restaurant service staff and improving the customer's dining experience.

[0096] In some embodiments, when a restaurant is equipped with multiple robots equipped with trays, a server can receive a call command from a user via a call bell and determine the corresponding recycling task. After determining the recycling task, the server can obtain the positions of each robot and, based on the distance (e.g., straight-line distance or distance) between each robot's position and a target recycling area, determine the robot closest to the target recycling area, use it as the target robot, and control the target robot to perform the recycling task.

[0097] Optionally, to ensure efficient response, upon receiving a call command, the server may obtain a third test image taken of each robot, first determine a deployable robot based on each third test image, and then determine a target robot from among the deployable robots. The deployable robot may be a robot with any pallet layer in an empty or partially empty state.

[0098] In an embodiment of the present application, a call bell is set at each dining table so that the user can call the robot to recycle tableware. At the same time, after the user issues a call instruction through the call bell, the target recycling area can be accurately determined based on the area where the dining table that issued the call bell is located, and the corresponding recycling task can be generated. There is no need to manually enter the table number, so that the robot can accurately go to the corresponding dining table to recycle tableware based on the recycling task.

[0099] In some embodiments, the above step S102 includes:

[0100] The latest first image to be tested is used as the input of the pre-trained classification model to obtain the classification result output by the classification model. The classification result is used to indicate the load status of each layer of the pallet, which includes a full load state, an empty state, and a semi-empty state.

[0101] Specifically, when determining the loading status of the tray based on the first image to be tested, the loading status of the tray may be detected by classification detection, and then whether the emptying task is completed may be determined based on the classification result obtained by the classification detection.

[0102] To improve detection efficiency, the constructed classification model can be pre-trained to obtain a pre-trained classification model. During classification detection, the latest first image to be tested is directly input into the pre-trained classification model. The classification model classifies the loading status of each layer of pallets in the first image to be tested, and obtains a classification result that can indicate the loading status of each layer of pallets.

[0103] Among them, because the tray's load state is divided into two states: full state and empty state, even if the tray only carries a small amount of tableware, it will be classified as full by the classification model. At this time, the robot will not respond to the customer's call instructions in real time (but in fact, the tray can still carry new tableware), which greatly reduces the user experience and the robot's work efficiency. Therefore, in the embodiment of the present application, the tray's load state is divided into three categories: full state, semi-empty state, and empty state. If there are too many categories, the robot's computing and processing capabilities will be high, which may easily cause the robot to get stuck and increase the probability of misjudgment, affecting the efficiency and accuracy of the robot's operation.

[0104] In an embodiment of the present application, the load status of the pallet in the first test image is classified by a pre-trained classification model, and a classification result that can indicate the load status of each layer of the pallet is obtained, which can ensure classification efficiency and accuracy; at the same time, when classifying the load status of the pallet, the classification model includes not only the full load status and the empty status, but also the semi-empty status, so that when necessary, the robot with a pallet load status of the semi-empty status can respond to the user's call in real time, thereby improving the user experience.

[0105] In some embodiments, the default specifications of the images captured by the camera installed on the robot (such as 640*480) can be used as the specifications of the sample images, and the classification model can be trained using the sample images with the default resolution of the camera without cropping the images according to the standard specifications of the model. This enables the pre-trained classification model to extract more semantic information from the images when performing classification detection on the images captured by the camera, thereby improving the accuracy and precision of the classification.

[0106] When obtaining sample images for the classification model, a camera can be used to capture images of trays on site (such as trays full of tableware, half-full trays, and empty trays under different lighting conditions), and the captured images can be manually screened and classified to obtain labels corresponding to each image, thereby obtaining labeled sample images.

[0107] Since the light intensity in different restaurants may vary greatly, in order to address the impact of different light and shadows, when training the classification model based on sample images, data enhancement methods such as color jittering can be used to enhance the sample images. Small random changes can be made to the color of the sample images (such as brightness, contrast, and saturation) to increase the diversity of the sample images, so that the trained classification model can better adapt to different lighting conditions and color changes.

[0108] When training a classification model using acquired sample images, multiple classification models can be trained simultaneously, and the resulting models can be integrated to improve robustness and generalization. Alternatively, ensemble learning methods such as voting, averaging, and stacking can be used to combine the classification results of multiple classification models to improve classification accuracy.

[0109] In some embodiments, in order to reduce the computing power requirements of the robot, the required classification model can be built based on the YOLO model. The YOLO model has the characteristics of being fast, lightweight, and easy to deploy, and can be well applied to mobile robots.

[0110] Among them, in order to improve the accuracy of detection, the CSP module in the YOLO model can be replaced with the C2F module. The YOLO model based on the C2F module can extract feature information of different scales in the image, so that the constructed classification model can better detect whether there are objects of different sizes such as spoons and plates on the tray, thereby improving the robustness of the classification model and the accuracy of the classification results.

[0111] In other embodiments, when constructing a classification model based on the YOLO model of the C2F module, the PRElU function (Parametrized ReLU, ReLU function with parameters) can also be used as the activation function of the classification model. Compared with the original ReLU function of the YOLO model, the PRElU function adds a learnable parameter. The learnable parameter can be dynamically adjusted according to the characteristics of the pallet's loading state, so that the classification model can better adapt to the classification requirements of the pallet's loading state, further improving the accuracy of the classification model.

[0112] In other embodiments, the parameters in the classification model may be quantized using a low-bit quantization method, that is, the 32-bit floating-point data of the classification model is mapped to a low-bit width (such as 1 bit, 2 bits, or 8 bits). For example, quantization may be performed using the INT8 quantization method, that is, 32-bit floating-point data such as weights in the classification model are mapped to 8-bit fixed-point integers, thereby converting the floating-point algorithm in the classification model into a fixed-point algorithm, reducing the size of the classification model while speeding up the inference speed of the classification model, thereby improving the work efficiency of the robot.

[0113] In some embodiments, the user can put the tableware to be recycled into a recycling box of preset specifications and uniform color, which is used to load the recycled tableware. For example, it can be a blue recycling box that is 30 cm long, 25 cm wide, and 15 cm high. The recycling box can be placed on a tray in advance, or it can be pre-set at the dining table. The user can first put the tableware to be recycled into the recycling box, and when the robot comes to recycle the tableware, the recycling box containing the tableware to be recycled can be directly placed on the robot's tray. Correspondingly, when acquiring a sample image, an image containing a recycling box placed on a tray is also acquired as a sample image. The status of the tableware in the recycling box can include a full state, an empty state, and a semi-empty state. Using a recycling box can make it easier for the robot to recycle more tableware and avoid the tableware from being scattered. Recycling boxes of the same color also facilitate improving the efficiency of model training and the accuracy of classification and recognition.

[0114] In some embodiments, the robot is configured with a delivery mode and a recycling mode. When no recycling box is detected, the robot defaults to delivery mode, and the camera performs object detection. When the robot's camera detects that a recycling box has been placed on the tray, the robot can prompt whether to enter recycling mode, and the camera detects the load status in recycling mode. When the robot's operating mode is switched based on manual instructions, the camera performs recycling box detection. When recycling boxes are detected on each layer, the robot cannot enter delivery mode, reducing food contamination by recycling boxes and ensuring the customer's dining experience. By detecting recycling boxes, the same robot's operating mode can be flexibly switched, improving the robot's operating efficiency and utilization rate while ensuring a good customer experience.

[0115] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0116] Example 2:

[0117] Corresponding to the robot control method described in the above embodiment, Figure 5 A structural block diagram of the control device of the robot provided in an embodiment of the present application is shown. For the sake of convenience of explanation, only the parts related to the embodiment of the present application are shown.

[0118] Reference Figure 6 The device includes: a first image acquisition module 61, a first judgment module 62 and a control module 63.

[0119] A first image acquisition module 61 is configured to capture each layer of the pallet when the robot reaches a preset cleaning area to obtain a first image to be tested, wherein the first image to be tested is updated at a preset frequency;

[0120] A first judgment module 62 is configured to judge the loading status of each layer of the pallet based on the latest first image to be tested, and obtain a first judgment result;

[0121] The control module 63 is used to control the robot according to the first judgment result.

[0122] In some embodiments, the control module 63 includes:

[0123] The control unit is used to execute a standby task when the latest first judgment result indicates that each layer of the pallets is in an empty state. The standby task is used to control the robot to go to a preset waiting area to wait for a call.

[0124] In an embodiment of the present application, when the robot reaches a preset cleaning area, it will capture each layer of trays provided on the robot to obtain a corresponding first image to be tested. The obtained first image to be tested can reflect the loading status of the tray. Therefore, the loading status of the tray can be actively determined based on the first image to determine whether the tableware on the tray has been emptied. Thus, when the first judgment result indicates that the tableware on the tray has been emptied, the robot can be controlled accordingly in a timely manner, without the need for the user to manually input commands to passively determine whether the tableware on the tray has been emptied. This reduces manual dependence while improving the robot's work efficiency. Furthermore, when acquiring the first image to be tested, the first image to be tested is updated at a preset frequency. Each judgment is based on the latest first image to be tested, which can ensure the real-time and accuracy of the first judgment result.

[0125] In some embodiments, the control device for controlling the robot further comprises:

[0126] The second image acquisition module is configured to capture each layer of the pallet to obtain a second image to be tested when the robot is located in the preset cleaning area and / or the preset waiting area and receives a call instruction.

[0127] The second judgment module is configured to judge the load status of each layer of the pallet based on the second image to be tested, and obtain a second judgment result.

[0128] The recycling control module is used to execute the recycling task corresponding to the above-mentioned call instruction when the above-mentioned second judgment result indicates that any layer of the above-mentioned tray is not fully loaded. The above-mentioned recycling task is used to control the above-mentioned robot to go to the target recycling area to recycle tableware.

[0129] In some embodiments, the control device of the robot further comprises:

[0130] A third image acquisition module for obtaining a test image is configured to photograph each layer of the pallet to obtain a third image for testing when it is determined that the recycling task is completed;

[0131] a third judgment module, configured to judge the load status of each layer of the pallet based on the third image to be tested, and obtain a third judgment result;

[0132] The emptying control module is used to perform an emptying task when the third judgment result indicates that each layer of the tray is fully loaded. The emptying task is used to control the robot to go to the preset cleaning area to empty the tableware.

[0133] In some embodiments, the call command is issued by a call bell installed on the dining table, and the control device of the robot further includes:

[0134] A target recycling area determination module, configured to determine the target recycling area based on the dining table corresponding to the call bell that issued the call instruction;

[0135] The recycling task determination module is used to determine the corresponding recycling task according to the target recycling area.

[0136] In some embodiments, the control device of the robot further comprises:

[0137] A classification module is used to use the latest first image to be tested as the input of a pre-trained classification model to obtain a classification result output by the classification model, wherein the classification result is used to indicate the load status of each layer of the pallet, and the load status includes a full load state, an empty state, and a semi-empty state.

[0138] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0139] Example 3:

[0140] Figure 7 This is a schematic diagram of the structure of a robot provided in one embodiment of the present application. Figure 7 As shown, the robot 7 of this embodiment includes: at least one processor 70 ( Figure 7 Only one processor is shown in the figure), a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, wherein the processor 70 implements the steps of any of the above-mentioned method embodiments when executing the computer program 72.

[0141] The robot 7 can be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The robot may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that Figure 7 The robot 7 is merely an example and does not constitute a limitation on the robot 7. The robot 7 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the robot 7 may also include input and output devices, network access devices, etc.

[0142] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0143] In some embodiments, the memory 71 may be an internal storage unit of the robot 7, such as a hard drive or memory of the robot 7. In other embodiments, the memory 71 may also be an external storage device of the robot 7, such as a plug-in hard drive, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the robot 7. Furthermore, the memory 71 may include both an internal storage unit of the robot 7 and an external storage device. The memory 71 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 71 may also be used to temporarily store data that has been output or is about to be output.

[0144] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0145] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0146] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0147] An embodiment of the present application provides a computer program product. When the computer program product runs on a robot, the robot can implement the steps in the above-mentioned various method embodiments when executing the computer program product.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / robot, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard drive, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0149] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0150] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0152] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0153] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A robot control method, characterized in that: The robot is provided with at least one tray for carrying tableware, and the method comprises: When the robot reaches a preset cleaning area, it photographs each layer of the pallet to obtain a first image to be tested, wherein the first image to be tested is updated at a preset frequency; Determine the load status of each layer of the pallet based on the latest first image to be tested, and obtain a first determination result; The robot is controlled according to the first judgment result.

2. The robot control method according to claim 1, wherein: The controlling the robot according to the first judgment result includes: When the latest first judgment result indicates that the trays on each layer are in an empty state, a standby task is executed, where the standby task is used to control the robot to go to a preset waiting area to wait for a call.

3. The robot control method according to claim 2, wherein: When the robot is located in a preset cleaning area, the method further includes: When receiving the clearing completion instruction input by the user, the standby task is executed.

4. The robot control method according to claim 3, wherein: The method further comprises: When the robot is located in the preset cleaning area and / or the preset waiting area, if a call instruction is received, the robot takes a picture of each layer of the pallet to obtain a second image to be measured; Determine the load status of each layer of the pallet based on the second image to be tested, and obtain a second determination result; When the second judgment result indicates that any layer of the tray is not fully loaded, the recycling task corresponding to the call instruction is executed, and the recycling task is used to control the robot to go to the target recycling area to recycle tableware.

5. The robot control method according to claim 4, wherein: After executing the recovery task corresponding to the call instruction, the method further includes: When it is determined that the recycling task is completed, photographing each layer of the pallet to obtain a third image to be tested; Determine the load status of each layer of the pallet based on the third image to be tested, and obtain a third determination result; When the third judgment result indicates that each layer of the tray is in a fully loaded state, an emptying task is performed, and the emptying task is used to control the robot to go to the preset cleaning area to empty the tableware.

6. The robot control method according to claim 4, wherein: The calling instruction is issued by a calling bell provided on the dining table. Before executing the recycling task corresponding to the calling instruction, the method further includes: Determining the target recycling area based on the dining table corresponding to the call bell that issued the call instruction; The corresponding recycling task is determined according to the target recycling area.

7. The robot control method according to any one of claims 1 to 6, characterized in that: The determining the load status of each layer of the pallet based on the latest first image to be measured to obtain a determination result includes: The latest first image to be tested is used as input of a pre-trained classification model to obtain a classification result output by the classification model, wherein the classification result is used to indicate the load status of each layer of the pallet, and the load status includes a full load status, an empty state, and a semi-empty state.

8. A robot control device, characterized in that: The robot is provided with at least one tray for carrying tableware, and the device comprises: a first image to be tested acquisition module, configured to photograph each layer of the pallet when the robot reaches a preset cleaning area to obtain a first image to be tested, wherein the first image to be tested is updated at a preset frequency; a first judgment module, configured to judge the load status of each layer of the pallet based on the latest first image to be tested, and obtain a first judgment result; A control module is used to control the robot according to the first judgment result.

9. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images