Workpiece material supplementing information processing method, device and equipment and storage medium

Abstract

The invention provides a workpiece supplementing information processing method and device, equipment and a storage medium, and the method comprises the steps: displaying a first interface when a supplementing confirmation message is received; the first interface comprises first information and a first control, the first information is graphic and text information associated with the missing target workpiece, and the first control is an option for confirming feeding; in response to an operation of triggering the first control by the user, sending an indication message to a server, the indication message being used for indicating the server to send a material supplementing instruction to the target material supplementing robot; and when a confirmation message from the server is received, a second interface is displayed, the second interface comprises second information which is generated according to the confirmation message and used for prompting completion of material supplementing, and the confirmation message is a message generated according to the first image collected by the humanoid robot and the second image collected by the target material supplementing robot. According to the method, the robot is confirmed to supplement materials at the terminal equipment side, and the humanoid robot and the material supplementing robot cooperate to confirm that the material supplementing is completed, so that the material supplementing efficiency is improved.

Description

Technical Field

[0001] This application relates to the technical field of robots, and particularly to a method, device, equipment and storage medium for processing workpiece replenishment information. Background Art

[0002] With the development of society, robots are widely used in various fields, such as industrial manufacturing and logistics transportation, etc., to further improve production efficiency. However, in the prior art, for example, when performing the task of sorting workpieces in a workpiece warehouse in the industrial manufacturing field, when a robot is working, it independently performs operations such as workpiece sorting, replenishment, and loading, and the replenishment efficiency is low. Summary of the Invention

[0003] This application provides a method, device, equipment and storage medium for processing workpiece replenishment information. Among them, the method can instruct a robot to replenish materials through a terminal device, and cooperate with a humanoid robot and a replenishment robot to confirm the completion of replenishment, so as to improve the replenishment efficiency.

[0004] In a first aspect, an embodiment of this application provides a method for processing workpiece replenishment information, which is applied to a terminal device of a robot system. The robot system includes the terminal device, a server connected to the terminal device, a humanoid robot connected to the server, and at least one replenishment robot. The method includes: When receiving a replenishment confirmation message, display a first interface; the first interface includes first information and a first control. The first information is graphic and text information related to a missing target workpiece generated according to the replenishment confirmation message, and the first control is an option for confirming to perform replenishment; In response to an operation by a user to trigger the first control within the first interface, send an instruction message to the server. The instruction message is used to instruct the server to send a replenishment instruction to a target replenishment robot to control the target replenishment robot to perform replenishment. The target replenishment robot is a robot in an idle state among the at least one replenishment robot; When receiving a confirmation message from the server, display a second interface. The second interface includes second information. The second information is graphic and text information for prompting the completion of replenishment generated according to the confirmation message. The confirmation message is a message forwarded by the server from the humanoid robot. The confirmation message is a message generated according to a first image and a second image. The first image is an image collected by the humanoid robot after receiving a replenishment completion message and including the workpiece placed on the workbench by the target replenishment robot. The replenishment completion message includes the second image. The second image is an image collected by the target replenishment robot when placing the target workpiece on the workbench according to the replenishment instruction and including the target workpiece.

[0005] In a second aspect, an embodiment of the present application provides a processing device for workpiece replenishment information, which is applied to a terminal device of a robot system. The robot system includes the terminal device, a server connected to the terminal device, a humanoid robot connected to the server, and at least one replenishment robot. The device includes: A first display unit, which displays a first interface when receiving a replenishment confirmation message. The first interface includes first information and a first control. The first information is graphic and text information related to the missing target workpiece generated according to the replenishment confirmation message, and the first control is an option for confirming replenishment. A sending unit, which is configured to send an instruction message to the server in response to an operation by the user to trigger the first control within the first interface. The instruction message is used to instruct the server to send a replenishment instruction to a target replenishment robot to control the target replenishment robot to perform replenishment. The target replenishment robot is a robot in an idle state among the at least one replenishment robot. A second display unit, which is configured to display a second interface when receiving a confirmation message from the server. The second interface includes second information, which is graphic and text information for prompting that replenishment is completed generated according to the confirmation message. The confirmation message is a message forwarded by the server from the humanoid robot, and the confirmation message is a message generated according to a first image and a second image. The first image is an image collected by the humanoid robot after receiving a replenishment completion message and including the workpiece placed on the workbench by the target replenishment robot. The replenishment completion message includes the second image, and the second image is an image collected by the target replenishment robot when placing the target workpiece on the workbench according to the replenishment instruction and including the target workpiece.

[0006] In a third aspect, an embodiment of the present application provides a terminal device, which includes at least one processor, a communication interface, and a memory. The communication interface is used to send and / or receive data, the memory is used to store a computer program, and the at least one processor is used to call the computer program stored in the memory to implement any method in the first aspect of the present application.

[0007] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the instructions for the steps in any method in the first aspect of the present application.

[0008] Fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute some or all of the steps described in any method of the first aspect of the embodiments of the present application.

[0009] Sixth aspect, the present application provides a computer program, wherein the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiments of the present application. This computer program can be a software installation package.

[0010] It can be seen that in the embodiment of the present application, when a replenishment confirmation message is received, a first interface is displayed. The first interface includes first information and a first control. The first information is graphic and text information related to the missing target workpiece generated according to the replenishment confirmation message, and the first control is an option to confirm replenishment. In response to an operation by the user to trigger the first control within the first interface, an indication message is sent to the server. The indication message is used to instruct the server to send a replenishment instruction to the target replenishment robot to control the target replenishment robot to perform replenishment. The target replenishment robot is a robot in an idle state among at least one replenishment robot.

[0011] When a confirmation message from the server is received, a second interface is displayed. The second interface includes second information. The second information is graphic and text information generated according to the confirmation message to prompt that replenishment is completed. The confirmation message is a message forwarded by the server from the humanoid robot. The confirmation message is a message generated according to a first image and a second image. The first image is an image collected by the humanoid robot after receiving the replenishment completion message and including the workpiece placed on the workbench by the target replenishment robot. The replenishment completion message includes the second image. The second image is an image collected by the target replenishment robot when placing the target workpiece on the workbench according to the replenishment instruction and including the target workpiece. In the present application, a terminal device can be used to instruct a robot to perform a replenishment operation, improving the interaction efficiency, and based on the cooperation between the humanoid robot and the replenishment robot to confirm the completion of replenishment, improving the replenishment efficiency. Description of the Drawings

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0013] Figure 1 It is a schematic structural diagram of a robot system provided by an embodiment of the present application; Figure 2 A schematic flowchart of a method for processing workpiece replenishment information provided by an embodiment of the present application; Figure 3 A schematic diagram of a first display interface provided by an embodiment of the present application; Figure 4 Another schematic diagram of a first display interface provided by an embodiment of the present application; Figure 5 A schematic diagram of a second display interface provided by an embodiment of the present application; Figure 6 A schematic diagram of a humanoid robot picking up a single target workpiece of a target replenishment robot provided by an embodiment of the present application; Figure 7 A functional unit composition block diagram of a device for processing workpiece replenishment information provided by an embodiment of the present application; Figure 8 A schematic structural diagram of a server provided by an embodiment of the present application. Detailed implementation manners

[0014] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present application.

[0015] The terms "first", "second", etc. in the specification and claims of the present application and the above accompanying drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products, or devices.

[0016] Referring to

[0017] Please refer to Figure 1 , Figure 1The structural schematic diagram of a robot system provided by an embodiment of the present application is as follows. As Figure 1 shown, the robot system includes a terminal device, a server, a humanoid robot, and at least one feeding robot. The server is respectively connected to the terminal device, the humanoid robot, and at least one feeding robot, so as to facilitate data interaction between the server and the terminal device, the humanoid robot, and at least one feeding robot.

[0018] It can be understood that the number of feeding robots in at least one feeding robot can be set according to actual needs. For example, two or more can be connected, and specifically, it is not limited here. Moreover, the feeding robot in at least one feeding robot can also be a humanoid robot, and the type of the feeding robot is not limited here.

[0019] The terminal device in the present application can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a wearable device, a head-mounted device, a vehicle-mounted terminal, etc. installed with a control application (i.e., an application client). The types of the terminal device are not specifically limited here.

[0020] It should be understood that when the application client runs on the terminal device, it can perform data interaction with Figure 1 the server shown.

[0021] Next, the method in the embodiment of the present application will be introduced in detail in conjunction with Figure 2 : Please refer to Figure 2 , Figure 2 which is the flowchart of a method for processing workpiece feeding information provided by an embodiment of the present application.

[0022] As Figure 2 shown, a method for processing workpiece feeding information, which is applied to the terminal device of the robot system. The robot system includes the terminal device, a server connected to the terminal device, a humanoid robot connected to the server, and at least one feeding robot. The method includes; S201, when receiving a feeding confirmation message, display a first interface.

[0023] Among them, the first interface includes first information and a first control. The first information is graphic and text information related to the missing target workpiece generated according to the feeding confirmation message, and the first control is an option for confirming to perform feeding.

[0024] Specifically, please refer to Figure 3 , Figure 3 which is the schematic diagram of a first display interface provided by an embodiment of the present application.

[0025] When the terminal device receives the replenishment confirmation message sent by the server, it displays the first interface, as Figure 3 shown. The first interface includes a first display area 31 and a second display area 32. The first display area 31 is used to display the fixed icons of the terminal device. For example, the time icon XX:XX on the left side of the first display area 31, and the battery icon on the right side of the first display area 31 for characterizing the battery power. It can be understood that the fixed icons in the first display area 31 can be set according to actual needs and are not limited here.

[0026] The second display area 32 includes first information, a first control 324, and a second control 325. Among them, the first information includes a first sub-information 321, a second sub-information 322, and a third sub-information 323. The specific content of the first sub-information 321 is: Hello, I am a robot. It is used to indicate the identity of the message sender to the user on the terminal device side, thereby improving the user experience. It can be understood that the number of sub-informations included in the second display area 32 can be set according to actual needs and is not limited here.

[0027] For example, the specific content of the second sub-information 322 can be: An image associated with the missing target workpiece. Among them, the image includes a workbench and a material box lacking workpieces on the workbench. It is convenient for the user on the terminal device side to detect whether there is a shortage of materials without the user having to go to detect by themselves, improving the interaction efficiency. The specific content of the third sub-information 323 can be: Hello, I see that the target part is missing. Please replenish the material. It is used to prompt the user to replenish the material. Among them, the first information indicates the material shortage in a way that combines images and text, making the first information more credible and improving the interaction efficiency. And in the second display area 32, there is also a prompt statement "Confirm to replenish the material", which is used to prompt the user whether to replenish the material. If the user agrees to replenish the material, the first control 324 can be triggered. If the user refuses to replenish the material, the second control 325 can be triggered. Thus, there is no need for the user to manually input, improving the operation convenience and further improving the user experience. It can be understood that the text content in the first information can be set according to actual needs. For example, it can also include text information such as the specific model of the target workpiece, and the specific content is not limited here. To further facilitate the user to confirm and improve the confirmation efficiency.

[0028] In a possible example, after the user triggers different controls, corresponding update information can be generated. The update information includes the text content of the user's confirmation of replenishment or refusal of replenishment, and this update information is used to update the first interface.

[0029] For example, please refer to Figure 4 , Figure 4 which is a schematic diagram of another first display interface provided by the embodiment of the present application.

[0030] As Figure 4As shown, if the user triggers the first control 324, first update information is generated. After the first interface is updated with the first update information, as Figure 4 shown, the specific content of the first sub-information 321 in the second display area 32 is: Hello, I am a robot. The specific content of the second sub-information 322 is: An image associated with the missing target workpiece. The image includes a workbench and a material box lacking a workpiece on the workbench. The specific content of the third sub-information 323 can be: Hello, I see that the target part is missing. Please replenish the material. The prompt statement "Confirm to replenish the material" and the first control 324 and the second control 325 disappear, and the user confirmation content 326 is updated and displayed. The specific content of the user confirmation content 326 is: Confirm to replenish the material. This facilitates the user to subsequently detect whether the replenishment is confirmed and improves the user experience.

[0031] S202. In response to an operation by the user to trigger the first control in the first interface, send an indication message to the server.

[0032] The indication message is used to instruct the server to send a replenishment instruction to the target replenishment robot to control the target replenishment robot to perform replenishment. The target replenishment robot is a robot in an idle state among the at least one replenishment robot.

[0033] Specifically, when the user triggers the first control, an indication message is sent to the server according to the user's operation in the first interface. After receiving the indication message, the server first obtains the to-be-executed work tasks of each replenishment robot among the at least one replenishment robot, and preferentially sets the replenishment robot with a to-be-executed work task amount of zero as the target replenishment robot. If there is no replenishment robot with a work task amount of zero, the replenishment robot with the least work task amount is preferentially set as the target replenishment robot, which improves the control intelligence and at the same time improves the replenishment efficiency. After determining the target replenishment robot, the server sends a replenishment instruction to the target replenishment robot to control the target replenishment robot to perform replenishment.

[0034] S203. When receiving a confirmation message from the server, display a second interface.

[0035] Among them, the second interface includes second information, and the second information is graphic information for prompting the completion of replenishing materials generated according to the confirmation message. The confirmation message is a message forwarded by the server from the humanoid robot, and the confirmation message is a message generated according to a first image and a second image. The first image is an image collected by the humanoid robot after receiving the replenishing material completion message and including the workpiece placed on the workbench by the target replenishing robot. The replenishing material completion message includes the second image, and the second image is an image collected by the target replenishing robot when placing the target workpiece on the workbench according to the replenishing instruction and including the target workpiece.

[0036] Specifically, please combine Figure 5 , Figure 5 which is a schematic diagram of a second display interface provided by an embodiment of the present application.

[0037] As Figure 5 shown, when the target replenishing robot places the target workpiece on the workbench according to the replenishing instruction, it collects a second image including the target workpiece and generates a replenishing material completion message including the second image, so as to provide data support for subsequent confirmation of whether the replenishing material is completed.

[0038] The server sends the received replenishing material completion message to the humanoid robot. The humanoid robot collects a second image of the workpiece placed on the workbench by the target replenishing robot after receiving the replenishing material completion message, and then generates a confirmation message based on the first image and the second image. When the terminal device side receives the confirmation message from the server, it displays the second interface.

[0039] As Figure 5 shown, the second interface includes a third display area 41 and a fourth display area 42. The third display area 41 is used to display the fixed icons of the terminal device. For example, the time icon XX:XX on the left side of the third display area 41 and the battery icon representing the battery power on the right side of the third display area 41. It can be understood that the fixed icons in the third display area 41 can be set according to actual needs and are not limited here.

[0040] The fourth display area 42 displays the second piece of information. The second piece of information includes a fourth sub-information 421, a fifth sub-information 422, and a sixth sub-information 423, thereby prompting the user that the replenishment is completed. The content of the fourth sub-information 421 is: Hello, I am a robot. It is used to prompt the source of the message. The specific content of the fifth sub-information 422 is: an image associated with the replenished target workpiece. Among them, the image includes a workbench, a manipulator, and a replenishment box located on the workbench and including the target workpiece. It is convenient for the user on the terminal device side to confirm whether the replenishment is completed, without the user having to go for inspection by themselves, improving the interaction efficiency. The specific content of the sixth sub-information 423 is: Hello, the target part has been replenished, and sorting starts. It is used to prompt the user that the replenishment is completed and the subsequent operation starts.

[0041] It can be seen that in this example, the user can learn about the material shortage through the terminal device side and can instruct the robot to replenish the material, improving the interaction efficiency. And based on the cooperation between the humanoid robot and the replenishment robot to confirm the completion of the replenishment, while improving the accuracy of the confirmation result, the replenishment efficiency is also improved.

[0042] In a possible example, the replenishment confirmation message is obtained through the following steps: scanning, by the humanoid robot, a plurality of material boxes located on the workbench to obtain workpiece identification information of the workpieces in the plurality of material boxes; determining the target workpiece and the target identification of the target workpiece according to the workpiece identification information and a workpiece set, where the workpiece set includes a corresponding relationship between the required workpieces and the identifications preset; obtaining, according to the target identification, basic information of the target workpiece from a preset database, where the basic information includes a third image of the target workpiece; generating the replenishment confirmation message according to the basic information.

[0043] In a specific example, each workpiece has a corresponding identification. The humanoid robot is located at the position where the workbench is located, and scans the workpieces in a plurality of material boxes located on the workbench through the humanoid robot, so as to obtain the workpiece identification information of the workpieces in the plurality of material boxes. Obtain a workpiece set, where the workpiece set includes the workpiece identifications of the workpieces required for the operation. According to the workpiece identification information and the workpiece set, find the workpiece identifications that do not appear in the workpiece identification information, so as to determine the workpiece corresponding to the workpiece identification that does not appear in the workpiece identification information as the target workpiece. Then obtain the basic information of the target workpiece from the preset database according to the target identification, where the basic information includes a third image of the target workpiece, and finally generate a replenishment confirmation message according to the basic information, so that the replenishment confirmation message includes the third image of the target workpiece.

[0044] It can be seen that in this example, based on the humanoid robot scanning to confirm the workpieces existing on the workbench, and based on the existing workpieces and the required workpieces in the workpiece set to determine the target workpiece, the accuracy of the determined target workpiece is improved, and further the credibility of the replenishment confirmation message is improved.

[0045] In a possible example, please refer to Figure 6 , Figure 6 which is a schematic diagram of a humanoid robot picking up a single target workpiece from a target replenishment robot provided in an embodiment of the present application.

[0046] As Figure 6 shown, the replenishment instruction includes the target identifier, the third image, and the position information of the workbench. The replenishment instruction is used to instruct the target replenishment robot to obtain the target number information of the target material rack for storing the target workpiece from a preset workpiece database according to the target identifier. The workpiece database includes the correspondence between the identifier of the workpiece and the number information of the material rack storing the workpiece; instruct the target replenishment robot to confirm the target position of the target workpiece on the target material rack according to the target number information and the third image; instruct the target replenishment robot to transfer a preset number of the target workpieces from the target position to the replenishment box; instruct the target replenishment robot to move the replenishment box to the workbench corresponding to the position information; when placing the replenishment box on the workbench by the first manipulator 53 of the target replenishment robot, instruct the second manipulator 52 of the target replenishment robot to grab a single target workpiece in the replenishment box to the first reference position, and instruct the target replenishment robot to collect the second image. The first reference position is determined according to the position of the target replenishment robot, the position of the humanoid robot, the arm length value of the target replenishment robot, and the arm length value of the humanoid robot. Among them, the replenishment instruction is further used to: instruct the target replenishment robot to obtain the position of the material rack corresponding to the target number information; instruct the target replenishment robot to move to the position of the material rack; instruct the target replenishment robot to collect the fourth image of the material rack at the position of the material rack; instruct the target replenishment robot to determine the target position of the target workpiece on the target material rack according to the fourth image and the third image.

[0047] In a specific example, in response to an operation of the user triggering the first control in the first interface, the terminal device sends an indication message to the server. After receiving the indication message, the server sends a replenishment instruction to the target replenishment robot. The target replenishment robot performs the following operations under the instruction of the replenishment instruction: The target replenishment robot searches the preset workpiece database for the target number information corresponding to the target identifier. The target number information is the number information of the target material rack for storing the target workpiece. Then the target replenishment robot confirms the target position of the target workpiece on the target material rack according to the number information and the third image.

[0048] Specifically, the target feeding robot can query the position of the material rack corresponding to the target number information from the database. The target feeding robot moves to the position of the material rack to facilitate obtaining the target workpiece. After the target feeding robot moves to the position of the material rack, a fourth image of the material rack at the material rack position is collected. Then, based on the fourth image and the third image, the target position of the target workpiece on the target material rack is determined. This makes the determined target position more credible, thereby improving the accuracy of the target feeding robot when grasping.

[0049] After confirming the target position, the target feeding robot transfers a preset number of target workpieces from the target position to the feeding box, and then moves the feeding box to the workbench corresponding to the position information. Among them, as Figure 6 shown, when the target feeding robot places the feeding box on the workbench, the target feeding robot uses the first manipulator 53 to place the feeding box on the workbench, uses the second manipulator 52 to grasp a single target workpiece in the feeding box to the first reference position, and at the same time the target feeding robot collects a second image.

[0050] It can be seen that in this example, the target feeding robot uses one hand to place the feeding box on the workbench and uses the other hand to move a single workpiece to the first reference position, so as to directly feed the humanoid robot, avoiding the humanoid robot waiting to pick up the workpiece from the feeding box after the feeding box is placed, making the various links of feeding more coordinated and improving the efficiency.

[0051] In a possible example, the first reference position is determined according to the following steps: obtaining the position of the target feeding robot, the position of the humanoid robot, the arm length value of the target feeding robot, and the arm length value of the humanoid robot through the target feeding robot; controlling the target feeding robot to obtain a first distance value between the target feeding robot and the humanoid robot according to the position of the target feeding robot and the position of the humanoid robot; if the sum of the arm length value of the target feeding robot and the arm length value of the humanoid robot is greater than or equal to the first distance value, then obtaining a first difference value obtained by subtracting the arm length value of the humanoid robot from the first distance value through the target feeding robot; controlling the target feeding robot to obtain the first reference position according to the first difference value, the position of the humanoid robot, and the position of the target feeding robot.

[0052] In a specific example, the target feeding robot obtains the location of the target feeding robot, the location of the humanoid robot, the arm length value of the target feeding robot, and the arm length value of the humanoid robot. The target feeding robot obtains a first distance value between the target feeding robot and the humanoid robot based on the location of the target feeding robot and the location of the humanoid robot. Calculate the sum of the arm length value of the target feeding robot and the arm length value of the humanoid robot. If the sum is greater than or equal to the first distance value, it means that the target feeding robot and the humanoid robot can transfer a single target workpiece. If the sum is less than the first distance value, the target feeding robot needs to move in the direction of the humanoid robot. Specifically, calculate the difference between the sum and the first distance value, and control the target feeding robot to move a distance value equal to the difference in the direction of the humanoid robot.

[0053] If it is determined that the sum of the arm length value of the target feeding robot and the arm length value of the humanoid robot is greater than or equal to the first distance value, then obtain a first difference value obtained by subtracting the arm length value of the humanoid robot from the first distance value through the target feeding robot. Then, a distance value greater than the first difference value but less than the arm length value of the target feeding robot can be randomly determined, and a first reference position is determined based on the location of the target feeding robot, the location of the humanoid robot, and the distance value.

[0054] It can be seen that in this example, after determining the first distance value between the target feeding robot and the humanoid robot, the first reference position is determined by combining the arm length value of the target feeding robot and the arm length value of the humanoid robot, making the determined first reference position more reasonable, thereby improving the feeding efficiency.

[0055] In a possible example, please refer to again Figure 6 , the confirmation message is generated according to the following steps: when the humanoid robot receives the feeding completion message, collect the first image through the humanoid robot; obtain the second image in the feeding completion message through the humanoid robot; confirm whether the target feeding robot places the target workpiece on the workbench based on the first image and the second image through the humanoid robot; if it is confirmed that the target feeding robot places the target workpiece on the workbench, then control the third manipulator 51 on one side of the humanoid robot close to the target feeding robot to move to a second reference position through the humanoid robot, and obtain a single target workpiece on the second manipulator 52 through the third manipulator 51 of the humanoid robot. The second reference position is determined according to the location of the humanoid robot, the arm length value of the humanoid robot, the first reference position, and the workpiece length value of the target workpiece. The distance value between the second reference position and the first reference position is less than the workpiece length value; when it is monitored that the third manipulator 51 obtains a single target workpiece on the second manipulator 52, generate the confirmation message through the humanoid robot.

[0056] In a specific example, the replenishment completion message includes a second image. When the humanoid robot receives the replenishment completion message from the target replenishment robot forwarded by the server, the humanoid robot acquires a first image and obtains the second image in the replenishment completion message. The humanoid robot detects whether the replenishment box, the workbench, and the workpiece are included in the first image. And, it detects whether the replenishment box, the workbench, the target workpiece, and the manipulator picking up the replenishment box are included in the second image. If the first image includes the replenishment box, the workbench, and the workpiece, and the second image includes the replenishment box, the workbench, the target workpiece, and the manipulator picking up the replenishment box. Then, it further detects whether the workpiece in the first image is the same as the target workpiece in the second image. If they are the same, it is confirmed that the target replenishment robot places the target workpiece on the workbench. If it is confirmed that the target replenishment robot places the target workpiece on the workbench, the third manipulator 51 on the side of the humanoid robot close to the target replenishment robot is controlled by the humanoid robot to move to the second reference position, and the single target workpiece on the second manipulator 52 is picked up by the third manipulator 51 of the humanoid robot.

[0057] It can be seen that in this example, the humanoid robot further determines that the target replenishment robot replenishes the target workpiece to the workbench, ensuring the accuracy of the placement position. And the humanoid robot picks up the single target workpiece of the target replenishment robot, avoiding waiting for the target replenishment robot to complete the placement and then obtaining the target workpiece from the replenishment box, and directly using it in the next task link in a timely manner, making the replenishment process connect smoothly and improving the efficiency.

[0058] In a possible example, the second reference position is determined according to the following steps: The humanoid robot obtains the position of the humanoid robot, the first reference position, and the workpiece length value of the target workpiece; the humanoid robot obtains the second distance value between the position of the humanoid robot and the first reference position; the humanoid robot is controlled to obtain the second reference position according to the position of the humanoid robot, the second distance value, and the workpiece length value.

[0059] In a specific example, a humanoid robot obtains the position of the humanoid robot, a first reference position, and the workpiece length value of a target workpiece. Calculate the second distance value between the position of the humanoid robot and the first reference position. To enable the humanoid robot to obtain a single target workpiece at the first reference position, the workpiece length value of the target workpiece needs to be considered. That is, determine a preset proportion of the workpiece length value, calculate the difference between the second distance value and the preset proportion of the workpiece length value, and determine a second reference position based on this difference and the position of the humanoid robot. For example, if the second distance value is 20 cm, the preset proportion is one-half, and the workpiece length value is 6 cm, the obtained one-half workpiece length value is 3 cm. Calculate the difference between the second distance value and the preset proportion of the workpiece length value as 17 cm. The second reference position is between the position of the humanoid robot and the first reference position, that is, starting from the position of the humanoid robot and at a distance of 17 cm from the position of the humanoid robot.

[0060] It can be seen that in this example, when the humanoid robot determines the second reference position, considering the workpiece length value of the target workpiece makes the determined second reference position more reasonable, thereby improving the efficiency of the humanoid robot in obtaining the target workpiece.

[0061] In a possible example, after the third manipulator 51 of the humanoid robot reaches the second reference position, an image including the third manipulator 51 and the second manipulator 52 is collected and analyzed to determine the height positions of the third manipulator 51 and the second manipulator 52. If the third manipulator 51 is placed below the second manipulator 52 of the target feeding robot, perform the action of picking up the target workpiece inside the second manipulator 52 to obtain the target workpiece. If the third manipulator 51 is above the second manipulator 52 of the target feeding robot, perform the action of grasping the target workpiece inside the second manipulator 52 to obtain the target workpiece. Improve the intelligence of the feeding process.

[0062] In a possible example, the humanoid robot is located at the position of the workbench and periodically scans the workpieces in multiple material boxes on the workbench to obtain the workpiece identification information of the workpieces in the multiple material boxes and the real-time quantity value of the workpieces in each material box. Obtain a workpiece set, where the workpiece set includes the workpiece identification of the workpieces required for the operation and the alarm quantity value corresponding to each workpiece identification. That is, when the real-time quantity of the workpiece corresponding to the workpiece identification is less than or equal to the alarm quantity value, determine the workpiece with the real-time quantity value less than or equal to the alarm quantity value as the target workpiece. Then obtain the basic information of the target workpiece from a preset database according to the target identification of the target workpiece, where the basic information includes the image of the target workpiece and the required quantity value. Finally, generate a replenishment confirmation message based on the basic information and the real-time quantity value, so that the replenishment confirmation message includes the image of the target workpiece and the quantity to be replenished.

[0063] When the terminal device receives the replenishment confirmation message, it displays a third interface, which includes third information and a first control. Among them, the third information may include an image associated with the target workpiece. The image includes a workbench and a material box of the target workpiece located on the workbench. This facilitates the user on the terminal device side to detect whether there is a shortage of materials, eliminating the need for the user to go for inspection by themselves and improving the interaction efficiency. The first control can be a control representing the confirmation of replenishment, so as to facilitate the user on the terminal device side to quickly confirm, improve the operation convenience, and thus improve the replenishment efficiency. If the user triggers the first control, the terminal device sends an instruction message to the server.

[0064] After receiving the instruction message, the server first obtains the to-be-executed work tasks of each replenishment robot among at least one replenishment robot, and preferentially sets the replenishment robot with zero to-be-executed work task volume as the target replenishment robot. After the server confirms the target replenishment robot, it sends a replenishment instruction to the target replenishment robot. The replenishment instruction is used to instruct the target replenishment robot to obtain the target number information of the target material rack for storing the target workpiece from the preset workpiece database according to the target identifier. The workpiece database includes the correspondence between the identifier of the workpiece and the number information of the material rack storing the workpiece. The target replenishment robot obtains the material rack position corresponding to the target number information according to the indication of the replenishment instruction, and then moves to the material rack position. The target replenishment robot captures an image of the material rack at the material rack position to determine the target position of the target workpiece on the target material rack based on the image of the material rack at the material rack position and the image of the target workpiece in the basic information.

[0065] Based on the indication of the replenishment instruction, after the target replenishment robot determines the target position of the target workpiece on the target material rack, it transfers the required quantity of the target workpiece from the target position to the replenishment box, and then moves the replenishment box to the workbench.

[0066] When the first manipulator 53 of the target replenishment robot places the replenishment box on the workbench, the target replenishment robot captures an image including the manipulator of the humanoid robot and detects whether there is a target workpiece on the manipulator in the image including the manipulator of the humanoid robot. If not, it grabs a single target workpiece in the replenishment box to the first reference position through the second manipulator 52 of the target replenishment robot. When the target replenishment robot places the target workpiece on the workbench according to the replenishment instruction, it captures a first acquisition image including the target workpiece and generates a replenishment completion message including the first acquisition image.

[0067] The server sends the received replenishment completion message to the humanoid robot. After receiving the replenishment completion message, the humanoid robot acquires a second acquisition image including the workpiece placed on the workbench by the target replenishment robot, and then generates a confirmation message based on the first acquisition image and the second acquisition image. Specifically, it is confirmed whether the workpiece in the first acquisition image and the second acquisition image is the target workpiece, and the first quantity of the target workpiece in the first acquisition image and the second quantity of the target workpiece in the second acquisition image are counted. When the first quantity is equal to the second quantity or the difference is 1, a confirmation message is generated. When the terminal device receives the confirmation message from the server, it displays the fourth interface.

[0068] It can be seen that in this example, replenishment is started when the number of workpieces is lower than the alarm quantity value, further improving the efficiency of replenishment.

[0069] The above mainly introduces the solution of the embodiment of the present application from the perspective of the execution process on the method side. It can be understood that in order for the electronic device to implement the above functions, it includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments provided in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0070] The embodiment of the present application can divide the functional units of the electronic device according to the above method examples. For example, each functional unit can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. It should be noted that the division of units in the embodiment of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0071] In the case of dividing each functional module corresponding to each function, the following is combined with Figure 7 A processing device for workpiece replenishment information in the embodiment of the present application is described in detail. Figure 7 It is a functional unit composition block diagram of a processing device for workpiece replenishment information provided by an embodiment of the present application. A processing device for workpiece replenishment information is applied to a terminal device of a robot system. The robot system includes the terminal device, a server connected to the terminal device, a humanoid robot connected to the server, and at least one replenishment robot. The device includes; The first display unit 601 displays a first interface when receiving a replenishment confirmation message; the first interface includes first information and a first control. The first information is graphic and text information related to the missing target workpiece generated according to the replenishment confirmation message, and the first control is an option to confirm replenishment. The sending unit 602 is configured to send an indication message to the server in response to an operation by the user to trigger the first control within the first interface. The indication message is used to instruct the server to send a replenishment instruction to a target replenishment robot to control the target replenishment robot to perform replenishment. The target replenishment robot is a robot in an idle state among the at least one replenishment robot. The second display unit 603 is configured to display a second interface when receiving a confirmation message from the server. The second interface includes second information, which is graphic and text information for prompting the completion of replenishment generated according to the confirmation message. The confirmation message is a message forwarded by the server from the humanoid robot, and the confirmation message is generated based on a first image and a second image. The first image is an image including the workpiece placed on the workbench by the target replenishment robot collected by the humanoid robot after receiving the replenishment completion message. The replenishment completion message includes the second image, and the second image is an image including the target workpiece collected by the target replenishment robot when placing the target workpiece on the workbench according to the replenishment instruction.

[0072] In a possible example, the device includes a first processing unit, and the first processing unit is configured to: scan a plurality of material boxes located on the workbench through the humanoid robot to obtain workpiece identification information of the workpieces in the plurality of material boxes; determine the target workpiece and the target identification of the target workpiece according to the workpiece identification information and a workpiece set, where the workpiece set includes a preset correspondence between required workpieces and their identifications; obtain basic information of the target workpiece from a preset database according to the target identification, where the basic information includes a third image of the target workpiece; and generate the replenishment confirmation message according to the basic information.

[0073] In a possible example, the replenishment instruction includes the target identifier, the third image, and the position information of the workbench. The replenishment instruction is used to instruct the target replenishment robot to obtain the target number information of the target material rack for storing the target workpiece from a preset workpiece database according to the target identifier. The workpiece database includes the correspondence between the identifier of the workpiece and the number information of the material rack storing the workpiece; instruct the target replenishment robot to confirm the target position of the target workpiece on the target material rack according to the target number information and the third image; instruct the target replenishment robot to transfer a preset number of the target workpieces from the target position to the replenishment box; instruct the target replenishment robot to move the replenishment box to the workbench corresponding to the position information; when placing the replenishment box on the workbench by the first manipulator of the target replenishment robot, instruct the second manipulator of the target replenishment robot to grab a single target workpiece in the replenishment box to the first reference position, and instruct the target replenishment robot to collect the second image. The first reference position is determined according to the position of the target replenishment robot, the position of the humanoid robot, the arm length value of the target replenishment robot, and the arm length value of the humanoid robot.

[0074] In a possible example, the replenishment instruction is further used to: instruct the target replenishment robot to obtain the position of the material rack corresponding to the target number information; instruct the target replenishment robot to move to the position of the material rack; instruct the target replenishment robot to collect a fourth image of the material rack at the position of the material rack; instruct the target replenishment robot to determine the target position of the target workpiece on the target material rack according to the fourth image and the third image.

[0075] In a possible example, the device includes a second processing unit, and the second processing unit is used to: obtain the position of the target replenishment robot, the position of the humanoid robot, the arm length value of the target replenishment robot, and the arm length value of the humanoid robot through the target replenishment robot; and control the target replenishment robot to obtain the first distance value between the target replenishment robot and the humanoid robot according to the position of the target replenishment robot and the position of the humanoid robot; and if the sum of the arm length value of the target replenishment robot and the arm length value of the humanoid robot is greater than or equal to the first distance value, obtain the first difference value obtained by subtracting the arm length value of the humanoid robot from the first distance value through the target replenishment robot; and control the target replenishment robot to obtain the first reference position according to the first difference value, the position of the humanoid robot, and the position of the target replenishment robot.

[0076] In a possible example, the device includes a third processing unit, which is configured to: when the humanoid robot receives the replenishment completion message, collect the first image through the humanoid robot; and obtain the second image in the replenishment completion message through the humanoid robot; and confirm whether the target replenishment robot places the target workpiece on the workbench based on the first image and the second image through the humanoid robot; and if it is confirmed that the target replenishment robot places the target workpiece on the workbench, control the third manipulator on the side of the humanoid robot close to the target replenishment robot to move to the second reference position through the humanoid robot, and obtain a single target workpiece on the second manipulator through the third manipulator of the humanoid robot, where the second reference position is determined according to the position of the humanoid robot, the arm length value of the humanoid robot, the first reference position, and the workpiece length value of the target workpiece, and the distance value between the second reference position and the first reference position is less than the workpiece length value; and when it is monitored that the third manipulator obtains a single target workpiece on the second manipulator, generate the confirmation message through the humanoid robot.

[0077] In a possible example, the third processing unit is further configured to: obtain the position of the humanoid robot, the first reference position, and the workpiece length value of the target workpiece through the humanoid robot; and obtain the second distance value between the position of the humanoid robot and the first reference position through the humanoid robot; and control the humanoid robot to obtain the second reference position according to the position of the humanoid robot, the second distance value, and the workpiece length value.

[0078] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of a server provided by an embodiment of the present application. As Figure 8 shown, the server includes a processor 701, a memory 703, a communication module 702, and a program 704. The processor 701 is communicatively connected to the memory 703 and the communication module 702 through an internal communication bus. It can be understood that the number of processors 701 can be set according to actual needs. For example, it can be one, two, or more, and the specific number of processors 701 is not limited here. The number of programs 704 can be set according to actual needs. For example, it can be one, two, or more, and the specific number of programs 704 is not limited here.

[0079] Among them, the program 704 is stored in the memory 703 and executed by the processor 701. The program 704 includes instructions for executing any step in the following method embodiments.

[0080] Among them, the processor 701 can be, for example, a Central Processing Unit (CPU), a general-purpose processor, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, units, and circuits described in connection with the disclosure of this application. The processor 701 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and so on. The communication unit can be a communication module 702, a transceiver, a transceiver circuit, etc., and the storage unit can be a memory 703.

[0081] The memory 703 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0082] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired or wireless manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more sets of available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0083] The embodiments of the present application further provide a computer storage medium. The computer storage medium stores a computer program for electronic data exchange, and the computer program causes the computer to execute some or all of the steps of any of the methods described in the above method embodiments. The above computer includes an electronic device.

[0084] The embodiments of the present application further provide a computer program product. The computer program product includes a computer program, and the computer program is operable to cause the computer to execute some or all of the steps of any of the methods described in the above method embodiments.

[0085] The computer program product can be a software installation package, and the above computer includes an electronic device.

[0086] It should be understood that in the various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0087] In several embodiments provided by the present application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the apparatuses or units can be in electrical, mechanical, or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can be physically included separately, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of a combination of hardware and software functional units.

[0090] The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software functional units stored in a storage medium include several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute some steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROM for short), random access memories (RAM for short), magnetic disks, or optical discs.

[0091] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions without departing from the spirit and scope of the present invention, and can make various modifications and alterations, including the combination of the above different functions and implementation steps, including software and hardware implementation manners, all within the protection scope of the present invention.

Claims

1. A method for processing workpiece replenishment information, characterized in that: A terminal device applied to a robot system, the robot system comprising the terminal device, a server connected to the terminal device, a humanoid robot connected to the server, and at least one feeding robot, the method comprising: When receiving the material replenishment confirmation message, a first interface is displayed; the first interface includes first information and a first control, the first information is graphic information associated with the missing target workpiece generated according to the material replenishment confirmation message, and the first control is an option for confirming material replenishment; In response to a user triggering an operation of the first control in the first interface, sending an instruction message to the server, wherein the instruction message is used to instruct the server to send a refilling instruction to a target refilling robot to control the target refilling robot to refill, wherein the target refilling robot is a robot in an idle state among the at least one refilling robot; When a confirmation message is received from the server, a second interface is displayed, the second interface including second information, the second information being graphic information generated according to the confirmation message to prompt that the material replenishment is completed, the confirmation message being a message from the humanoid robot forwarded by the server, the confirmation message being a message generated according to a first image and a second image, the first image being an image including a workpiece placed on the workbench by the target material replenishment robot and captured by the humanoid robot after receiving the material replenishment completion message, the material replenishment completion message including the second image, and the second image being an image including the target workpiece captured by the target material replenishment robot when placing the target workpiece on the workbench according to the material replenishment instruction.

2. The method according to claim 1, characterized in that The feeding confirmation message is obtained through the following steps: Scanning a plurality of material boxes on the workbench by the humanoid robot to obtain workpiece identification information of the workpieces in the plurality of material boxes; Determining the target workpiece and the target identification of the target workpiece according to the workpiece identification information and a workpiece set, wherein the workpiece set includes a preset correspondence relationship between required workpieces and identifications; Acquire basic information of the target workpiece from a preset database according to the target identifier, wherein the basic information includes a third image of the target workpiece; The feeding confirmation message is generated according to the basic information.

3. The method according to claim 2, characterized in that The material replenishment instruction includes the target identification, the third image and the position information of the workbench, and the material replenishment instruction is used to instruct the target material replenishment robot to obtain the target number information of the target material rack for storing the target workpiece from a preset workpiece database according to the target identification, and the workpiece database includes the corresponding relationship between the identification of the workpiece and the number information of the material rack storing the workpiece; Instructing the target material replenishing robot to confirm the target position of the target workpiece in the target material rack according to the target number information and the third image; Instructing the target material replenishing robot to transfer a preset number of the target workpieces from the target position to a material replenishing box; Instructing the target feeding robot to move the feeding box to the workbench corresponding to the position information; When the replenishing box is placed on the workbench by the first manipulator of the target replenishing robot, the second manipulator of the target replenishing robot is instructed to grab the single target workpiece in the replenishing box to the first reference position, and the target replenishing robot is instructed to collect the second image, wherein the first reference position is determined according to the position of the target replenishing robot, the position of the humanoid robot, the arm length of the target replenishing robot and the arm length of the humanoid robot.

4. The method according to claim 3, characterized in that The feeding instruction is also used for: Instructing the target feeding robot to obtain the material rack position corresponding to the target number information; Instructing the target feeding robot to move to the material rack position; Instructing the target feeding robot to collect a fourth image of the material rack at the material rack position; The target material replenishment robot is instructed to determine the target position of the target workpiece on the target material rack according to the fourth image and the third image.

5. The method according to claim 3, characterized in that: The first reference position is determined according to the following steps: Obtaining the location of the target feeding robot, the location of the humanoid robot, the arm length value of the target feeding robot, and the arm length value of the humanoid robot through the target feeding robot; Controlling the target feeding robot to obtain a first distance value between the target feeding robot and the humanoid robot according to the location of the target feeding robot and the location of the humanoid robot; If the sum of the arm length value of the target feeding robot and the arm length value of the humanoid robot is greater than or equal to the first distance value, a first difference value of the first distance value minus the arm length value of the humanoid robot is obtained by the target feeding robot; The target feeding robot is controlled to obtain the first reference position according to the first difference, the position of the humanoid robot and the position of the target feeding robot.

6. The method according to any one of claims 3 to 5, characterized in that: The confirmation message is generated according to the following steps: When the humanoid robot receives the feeding completion message, collecting the first image through the humanoid robot; Acquiring the second image in the feeding completion message by the humanoid robot; confirming, by the humanoid robot based on the first image and the second image, whether the target replenishing robot places the target workpiece on the workbench; If it is confirmed that the target feeding robot places the target workpiece on the workbench, the third manipulator of the humanoid robot close to the target feeding robot is controlled by the humanoid robot to move to a second reference position, and the single target workpiece on the second manipulator is obtained by the third manipulator of the humanoid robot, wherein the second reference position is determined according to the position of the humanoid robot, the arm length of the humanoid robot, the first reference position and the workpiece length value of the target workpiece, and the distance value between the second reference position and the first reference position is less than the workpiece length value; When it is monitored that the third manipulator obtains the single target workpiece on the second manipulator, the confirmation message is generated by the humanoid robot.

7. The method according to claim 6, characterized in that The second reference position is determined according to the following steps: Acquiring, by means of the humanoid robot, the position of the humanoid robot, the first reference position, and the workpiece length value of the target workpiece; Acquire, by the humanoid robot, a second distance value between the position of the humanoid robot and the first reference position; The humanoid robot is controlled to obtain the second reference position according to the position of the humanoid robot, the second distance value and the length value of the workpiece.

8. A device for processing workpiece replenishment information, characterized in that: A terminal device applied to a robot system, the robot system comprising the terminal device, a server connected to the terminal device, a humanoid robot connected to the server, and at least one feeding robot, the device comprising: A first display unit, when receiving a material replenishment confirmation message, displays a first interface; the first interface includes first information and a first control, the first information is graphic information associated with the missing target workpiece generated according to the material replenishment confirmation message, and the first control is an option for confirming material replenishment; a sending unit, configured to send an instruction message to the server in response to a user triggering an operation of the first control in the first interface, wherein the instruction message is used to instruct the server to send a refilling instruction to a target refilling robot to control the target refilling robot to refill, wherein the target refilling robot is a robot in an idle state among the at least one refilling robot; A second display unit is used to display a second interface upon receiving a confirmation message from the server, wherein the second interface includes second information, wherein the second information is graphic information generated according to the confirmation message to prompt that the material replenishment is completed, the confirmation message is a message from the humanoid robot forwarded by the server, and the confirmation message is a message generated according to the first image and the second image, the first image is an image including the workpiece placed on the workbench by the target material replenishment robot and captured by the humanoid robot after receiving the material replenishment completion message, the material replenishment completion message includes the second image, and the second image is an image including the target workpiece captured by the target material replenishment robot when placing the target workpiece on the workbench according to the material replenishment instruction.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images