Unstacking exception handling system

Abstract

The invention provides an unstacking exception handling system which comprises a processing unit, a display unit, an input unit and a sending unit, simulation software is loaded in the processing unit and used for building a simulation environment according to a real unstacking scene provided by a user, and the simulation environment is used for simulating the unstacking process in the real scene; when the robot vision system detects that the real environment is abnormal in the unstacking process, an unstacking abnormity prompt pops up in a simulation interface of the display unit; the processing unit receives the unstacking abnormity type fed back by the robot vision system; and according to the type of the unstacking exception, an unstacking exception handling option box pops up, and if the user selects that manual handling is needed, an exception handling task is issued to a worker in a real environment. Therefore, the unstacking process in the real environment can be monitored in real time through the virtual stack type in the simulation, so that a worker is reminded in time to deal with stacking abnormity in the dismounting process, and the real stacking process is tracked and managed conveniently.

Description

Background Art

[0002] Depalletizing is to grab the materials stacked on the pallet in a certain order and place them on the conveyor belt or material frame for storage or transportation. The automated depalletizing process usually involves identifying the location of the materials to be grabbed in the stack through a visual system, and then planning the robot's mechanical arm to grab and stack the materials.

[0003] At present, industrial robots are controlled to perform depalletizing operations through pre-made depalletizing programs. Since the pallet type is known, the robot will depalletize sequentially according to the pallet type during the entire operation process.

[0004] However, in real depalletizing scenarios, abnormal situations are prone to occur, such as the robot dropping the material it is grabbing, or hitting other materials in the stack during the grabbing process, causing the stack shape to change. At this time, if the robot cannot detect the depalletizing abnormality in time and handle it, it will cause the stack to collapse or become unstable, posing a safety hazard. Summary of the invention

[0005] In view of the defects in the prior art, an object of the present invention is to provide a depalletizing exception handling system.

[0006] In a first aspect, an embodiment of the present application provides a depalletizing exception handling system, comprising: a processing unit, a display unit, an input unit, and a sending unit, wherein the processing unit is loaded with simulation software for building a simulation environment according to a real depalletizing scene provided by a user, wherein the simulation environment is used to simulate the depalletizing process in a real scene; the depalletizing process in the simulation environment is in step with the depalletizing process in the real environment;

[0007] The processing unit is in communication with the robot vision system, and when the robot vision system detects that an abnormality occurs in the real environment during the depalletizing process, a depalletizing abnormality reminder pops up in the simulation interface of the display unit;

[0008] The processing unit receives the type of depalletizing anomaly fed back by the robot vision system;

[0009] The display unit is also used to pop up a depalletizing exception handling option box according to the type of depalletizing exception, so as to confirm whether to perform manual handling;

[0010] The input unit is used to receive the first trigger operation information input by the user. If the user selects that manual processing is required, the exception handling task is issued to the staff in the real environment through the sending unit.

[0011] Optionally, the robot vision system comprises: an image processor, and a plurality of cameras arranged around the depalletizing area and / or on the robot;

[0012] The camera transmits the collected image of the depalletizing area to the image processor;

[0013] The image processor analyzes the image of the depalletizing area to determine the type of depalletizing anomaly.

[0014] Optionally, the types of depalletizing exceptions include:

[0015] The material is still on the stack and the robot is judged to have failed to grab it;

[0016] The material is not on the stack, and the robot is judged to have dropped the material during the grasping process;

[0017] The material is on the conveyor or pallet, but the material is damaged.

[0018] Optionally, the input unit is further used to receive trigger indication information for the depalletizing exception processing option box, wherein the depalletizing exception processing option box includes: depalletizing exception type, and options for whether manual processing is required;

[0019] When receiving a trigger prompt message that requires manual processing, the display unit switches the current interface to the manual processing interface;

[0020] The input unit is also used to receive the quantity of materials that need to be manually processed, which is input by the user on the manual processing interface.

[0021] Optionally, the processing unit is specifically used to: if the robot fails to grasp and the material is not on the stack, instruct the staff to check whether the dropped material is damaged through the sending unit; if the material is not damaged, the staff will move the material to a conveyor belt or a pallet; if the material is damaged, instruct the staff to recycle the material or terminate the current depalletizing order through the sending unit;

[0022] The input unit is also used to receive the abnormality elimination indication information fed back by the staff;

[0023] After receiving the indication information that the abnormality has been eliminated, the processing unit increases the progress of the depalletizing task in the order information according to the number of materials manually processed.

[0024] Optionally, it also includes:

[0025] The depalletizing task progress detection unit is used to detect whether the depalletizing task progress in the order information increases according to a preset waiting period. If the depalletizing task progress increases, and the increased progress is consistent with the number of materials that need to be manually processed input on the manual processing interface, a prompt message indicating that the depalletizing exception processing has been completed is displayed through the display unit, and a start trigger indication message sent by the user is received through the input unit to continue to execute the unfinished depalletizing order;

[0026] If the depalletizing task progress does not increase, or the increased depalletizing task progress is inconsistent with the number of materials that need to be manually processed entered on the manual processing interface, the next waiting cycle is entered.

[0027] Optionally, it also includes:

[0028] A depalletizing step confirmation unit, used to determine whether the depalletizing process in the simulation environment is consistent with the depalletizing process in the real environment after displaying a prompt message that the depalletizing exception processing has been completed;

[0029] If they are inconsistent, the processing unit adjusts the stacking type in the simulation environment according to the stacking type in the real environment, so that the stacking type in the simulation environment is consistent with the stacking type in the real environment.

[0030] Optionally, it also includes:

[0031] The waiting timeout prompt unit is used to display a waiting timeout prompt box through the display unit when the number of waiting cycles reaches the upper limit threshold, so as to remind the staff to check the cause of the timeout.

[0032] Optionally, it also includes:

[0033] The storage unit is used to record the time spent on manually handling depalletizing exceptions, the number of times the exceptions are handled, and the information of the staff who handles each exception.

[0034] In a second aspect, an embodiment of the present application provides a robot, comprising a robot body, a drive system and a robotic arm, wherein the robot is used to perform a depalletizing operation according to a preset depalletizing rule, and a processor in the robot body, or a computer remotely connected to the robot body, is loaded with a depalletizing exception handling system as described in any one of the first aspects, and is used to pop up a depalletizing exception reminder in a simulation environment when an exception occurs during the depalletizing process in a real environment;

[0035] Determine the type of depalletizing anomaly through a visual system deployed in a real environment;

[0036] According to the type of depalletizing exception, a depalletizing exception handling option box pops up to confirm whether to perform manual processing;

[0037] If manual processing is required, issue exception handling tasks to staff in the real environment.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] In this application, a simulation environment is built according to the real depalletizing scene provided by the user, and the simulation environment is used to simulate the depalletizing process in the real scene; wherein, the depalletizing process in the simulation environment is in step with the depalletizing process in the real environment; when an abnormality occurs in the depalletizing process in the real environment, a depalletizing abnormality reminder pops up in the simulation environment; the type of depalletizing abnormality is determined by the visual system arranged in the real environment; according to the type of depalletizing abnormality, a depalletizing abnormality handling option box pops up to confirm whether manual processing is performed; if manual processing is required, an abnormality handling task is issued to the staff in the real environment. Thus, the depalletizing process in the real environment can be monitored in real time through the virtual stack type in the simulation, so that the user can complete the remote monitoring of the material disassembly in the simulation, and timely handle the stacking abnormality in the disassembly process, with strong reliability, and it is convenient to track and manage the real stacking process. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0041] Figure 1 Schematic diagram of the depalletizing exception handling interface based on HMI provided in the embodiment of the present application Figure 1 ;

[0042] Figure 2 Schematic diagram of the depalletizing exception handling interface based on HMI provided in the embodiment of the present application Figure 2 ;

[0043] Figure 3 Schematic diagram of the depalletizing exception handling interface based on HMI provided in the embodiment of the present application Figure 3 ;

[0044] Figure 4 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 1 ;

[0045] Figure 5 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 2 ;

[0046] Figure 6 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 3 ;

[0047] Figure 7 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 4 ;

[0048] Figure 8 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 5 ;

[0049] Fig. 9 A schematic diagram of the structure of a depalletizing abnormality processing device provided in one embodiment of the present application;

[0050] Fig.10 It is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there can be a central component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there can be a central component at the same time.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0054] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0056] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0057] Figure 1 Schematic diagram of the depalletizing exception handling interface based on HMI provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown in the figure, the depalletizing exception handling function is developed through the human machine interface (HMI), and a depalletizing virtual scene consistent with the real environment is constructed in the simulation interface. Figure 1 The depalletizing process in the simulation environment is demonstrated in the figure. When a depalletizing exception occurs, a depalletizing exception handling dialog box pops up in the simulation operation interface to confirm whether to manually place the box.

[0058] Figure 2 Schematic diagram of the depalletizing exception handling interface based on HMI provided in the embodiment of the present application Figure 2 ,like Figure 2 As shown, when Figure 1 In the dialog box shown, select the need for manual box placement processing, and the manual box placement processing dialog box will pop up. Enter the number of manual boxes in this dialog box.

[0059] Figure 3 Schematic diagram of the depalletizing exception handling interface based on HMI provided in the embodiment of the present application Figure 3 ,like Figure 3As shown in the figure, according to the real depalletizing scene provided by the user, a simulation environment is built, and robots, pallets, stacking, etc. are added to the simulation environment to simulate the depalletizing process in the real scene; and the depalletizing process in the simulation environment is ensured to be in step with the depalletizing process in the real environment. This is to enable remote monitoring of material disassembly in the HMI, and timely handle depalletizing anomalies during the disassembly process, with strong reliability, easy tracking and management of the real depalletizing process, and ensure the accuracy of depalletizing.

[0060] Figure 4 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 1 ,like Figure 4 As shown, the depalletizing exception handling system 400 in this embodiment may include: a processing unit 410, a display unit 420, an input unit 430, and a sending unit 440, wherein the processing unit 410 is loaded with simulation software for building a simulation environment according to the real depalletizing scene provided by the user, and the simulation environment is used to simulate the depalletizing process in the real scene; the depalletizing process in the simulation environment is in step with the depalletizing process in the real environment; the processing unit 410 is connected to the robot vision system for communication, and when the robot vision system detects that the real environment has an abnormality during the depalletizing process, a depalletizing exception reminder pops up in the simulation interface of the display unit 420; the processing unit 410 receives the type of depalletizing exception fed back by the robot vision system; the display unit 420 is also used to pop up a depalletizing exception handling option box according to the type of depalletizing exception to confirm whether manual processing is performed; the input unit 430 is used to receive the first trigger operation information input by the user. If the user chooses that manual processing is required, the exception handling task is issued to the staff in the real environment through the sending unit.

[0061] For example, the depalletizing process in the simulation environment is in step with the depalletizing process in the real environment. Figure 3 As shown, the simulation environment may include robots, conveyor belts, pallets, palletizing, etc., wherein the depalletizing progress on the pallet is consistent with the depalletizing progress in the real environment.

[0062] Exemplarily, the robot vision system includes: an image processor, and multiple cameras arranged around the depalletizing area and / or on the robot; the camera transmits the captured image of the depalletizing area to the image processor; the image processor analyzes the image of the depalletizing area to determine the type of depalletizing abnormality.

[0063] In this embodiment, the depalletizing process can be monitored in real time by a robot vision system deployed in a real environment. If a depalletizing abnormality occurs (for example, robot grasping failure, part drop, material damage, stack collapse, etc.), a depalletizing abnormality reminder will pop up in the simulation operation interface.

[0064] In this embodiment, the image of the depalletizing area can be collected by cameras arranged around the depalletizing area and / or on the robot; and the type of depalletizing abnormality is determined based on the image of the depalletizing area.

[0065] Exemplary types of depalletizing anomalies include the following:

[0066] The material is still on the stack, and the robot is judged to have failed to grab it;

[0067] The material is not on the stack, and it is determined that the robot has dropped the material during the grasping process;

[0068] The material is on the conveyor or pallet, but the material is damaged.

[0069] It should be noted that the present embodiment does not limit the type of exceptions, and the classification of exceptions may be increased according to various possibilities that may occur in a real environment, and different processing strategies may be set based on different exception types.

[0070] In this embodiment, after determining the type of depalletizing exception, a depalletizing exception handling option box pops up. Specifically, when the material is still on the stack, that is, it has not been grabbed by the robot (grasping failure), no manual processing is required at this time, and the robot is directly controlled to perform secondary grabbing.

[0071] In one possible situation, the robot fails to grab the material and drops it. In this case, select "manual handling required" in the exception handling option box that pops up on the HMI. The human will check whether the material is damaged. If not, the material will be moved to the conveyor belt or material frame (pallet). If it is damaged, it will be handled according to the handling plan for damaged materials in the operation manual, for example: recycling, or terminating the current depalletizing order.

[0072] In this embodiment, when manual processing is selected, a reminder is sent to the staff in the real environment, and the type of exception to be processed and the corresponding processing strategy are informed.

[0073] Exemplarily, the input unit 430 is also used to receive trigger indication information for a depalletizing exception handling option box, which includes: depalletizing exception type, and options for whether manual handling is required; when a trigger prompt information requiring manual handling is received, the display unit 420 switches the current interface to the manual handling interface; the input unit 430 is also used to receive the quantity of materials requiring manual handling entered by the user on the manual handling interface.

[0074] Exemplarily, the processing unit 410 is specifically used for: if the robot fails to grasp and the material is not on the stack, then the sending unit 440 is used to instruct the staff to check whether the dropped material is damaged. If the material is not damaged, the staff will move the material to the conveyor belt or pallet; if the material is damaged, then the sending unit 440 is used to instruct the staff to recycle the material or terminate the current depalletizing order; the input unit 430 is also used to receive the abnormality elimination indication information fed back by the staff; after receiving the abnormality elimination indication information, the processing unit 410 increases the depalletizing task progress in the order information according to the number of materials manually processed.

[0075] In this embodiment, if the robot fails to grab and the material is not on the stack, the staff is instructed to check whether the dropped material is damaged. If the material is not damaged, the staff will move the material to a conveyor belt or a pallet; the staff will receive the abnormality elimination indication information feedback from the staff, and the depalletizing task progress in the order information will be automatically increased according to the number of materials manually processed; if the material is damaged, the staff will recycle the material or terminate the current depalletizing order. In this way, the staff can be notified in time to handle the depalletizing abnormality and the progress of the abnormality handling can be monitored in a simulation environment.

[0076] In this embodiment, a simulation environment is built according to the real depalletizing scene provided by the user. When an abnormality occurs in the depalletizing process in the real environment, a depalletizing abnormality reminder pops up in the simulation environment; the type of depalletizing abnormality is determined by the visual system deployed in the real environment; according to the type of depalletizing abnormality, a depalletizing abnormality handling option box pops up to confirm whether manual processing is required; if manual processing is required, an abnormality handling task is issued to the staff in the real environment. Therefore, the depalletizing process in the real environment can be monitored in real time through the virtual stack type in the simulation, so that the user can complete the remote monitoring of material disassembly in the simulation, and timely handle the stacking abnormality in the disassembly process, which has strong reliability and is convenient for tracking and managing the real stacking process.

[0077] Figure 5 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 2 ,like Figure 5As shown, the depalletizing exception handling system 400 in this embodiment may include: a processing unit 410, a display unit 420, an input unit 430, a sending unit 440, and a depalletizing task progress detection unit 450, wherein the depalletizing task progress detection unit 450 is used to detect whether the depalletizing task progress in the order information increases according to a preset waiting period. If the depalletizing task progress increases, and the increased progress is consistent with the number of materials that need manual processing input on the manual processing interface, a prompt message indicating that the depalletizing exception handling has been completed is displayed through the display unit, and a start trigger indication message sent by the user is received through the input unit to continue executing the unfinished depalletizing order; if the depalletizing task progress does not increase, or the increased depalletizing task progress is inconsistent with the number of materials that need manual processing input on the manual processing interface, the next waiting period is entered.

[0078] In this embodiment, by detecting whether the progress of the depalletizing task in the order information increases according to the preset waiting cycle, if the progress of the depalletizing task increases, and the increased progress is consistent with the number of materials that need to be manually processed input on the manual processing interface, a prompt message indicating that the depalletizing exception processing has been completed is displayed, and the start trigger instruction information sent by the user is received, and the unfinished depalletizing order continues to be executed; if the progress of the depalletizing task does not increase, or the increased progress of the depalletizing task is inconsistent with the number of materials that need to be manually processed input on the manual processing interface, the next waiting cycle is entered. In this way, the progress of the depalletizing order can be monitored, and it is clear whether all the exceptions that need to be manually processed have been processed, ensuring that the depalletizing process in the simulation environment is in step with the depalletizing process in the real environment, so as to simulate the depalletizing process in the real environment through the simulation environment and achieve the purpose of remote monitoring.

[0079] Figure 6 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 3 ,like Figure 6 As shown, the depalletizing exception handling system 400 in this embodiment may include: a processing unit 410, a display unit 420, an input unit 430, a sending unit 440, a depalletizing task progress detection unit 450, and a depalletizing pace confirmation unit 460, wherein the depalletizing pace confirmation unit 460 is used to determine whether the pace of the depalletizing process in the simulation environment is consistent with that in the real environment after displaying a prompt message that the depalletizing exception handling has been completed; if not, the processing unit adjusts the stack type in the simulation environment according to the stack type in the real environment, so that the stack type in the simulation environment is consistent with the stack type in the real environment.

[0080] Figure 7 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 4 ,like Figure 7As shown, the depalletizing exception handling system 400 in this embodiment may include: a processing unit 410, a display unit 420, an input unit 430, a sending unit 440, a depalletizing task progress detection unit 450, a depalletizing pace confirmation unit 460, and a waiting timeout prompt unit 470, wherein the waiting timeout prompt unit 470 is used to display a waiting timeout prompt box through the display unit when the number of waiting cycles reaches an upper limit threshold, so as to remind the staff to check the cause of the timeout.

[0081] In this embodiment, the upper limit threshold of the number of waiting cycles is set to limit the time range of abnormal processing. When the number of waiting cycles reaches the upper and lower thresholds, the staff is notified to promptly investigate the cause of the timeout. If it is a normal situation (for example, the abnormal situation is complicated, the number of dropped materials is large, and the processing time does take a long time, the waiting cycle can be manually reset). If it is an abnormal situation, such as the abnormality has been eliminated, but the processing unit has not been notified in time to resume the execution of the order task, the input unit receives the start trigger instruction information sent by the user to continue to execute the unfinished depalletizing order, thereby ensuring the smooth progress of the depalletizing task and avoiding the robot from being stuck for too long.

[0082] Figure 8 A schematic diagram of a depalletizing exception handling system provided in an embodiment of the present application Figure 5 ,like Figure 8 As shown, the depalletizing exception handling system 400 in this embodiment may include: a processing unit 410, a display unit 420, an input unit 430, a sending unit 440, a depalletizing task progress detection unit 450, a depalletizing pace confirmation unit 460, a waiting timeout prompt unit 470, and a storage unit 480, wherein the storage unit 480 is used to record the time spent on manually handling depalletizing exceptions, the number of processing times, and the staff information of each exception handling.

[0083] In this embodiment, the storage unit records the time spent on manually handling depalletizing exceptions, the number of times the exceptions are handled, and the information of the staff who handles each exception, so that the probability of occurrence and processing time of various abnormal situations can be analyzed to facilitate the improvement of the robot's depalletizing program.

[0084] The present application also provides a method for handling depalletizing exceptions. The method in this embodiment may include:

[0085] Step S1: Building a simulation environment according to the real depalletizing scene provided by the user, where the simulation environment is used to simulate the depalletizing process in the real scene.

[0086] In this embodiment, the depalletizing process in the simulation environment is in step with the depalletizing process in the real environment. Figure 3As shown, the simulation environment may include robots, conveyor belts, pallets, palletizing, etc., wherein the depalletizing progress on the pallet is consistent with the depalletizing progress in the real environment.

[0087] Step S2: When an abnormality occurs in the depalletizing process in the real environment, a depalletizing abnormality reminder pops up in the simulation environment.

[0088] In this embodiment, the depalletizing process can be monitored in real time by a visual system deployed in a real environment. If a depalletizing abnormality occurs (for example, robot grasping failure, part drop, material damage, stack collapse, etc.), a depalletizing abnormality reminder will pop up in the simulation operation interface.

[0089] Step S3: determine the type of depalletizing anomaly through a visual system deployed in a real environment.

[0090] In this embodiment, the image of the depalletizing area can be collected by cameras arranged around the depalletizing area and / or on the robot; and the type of depalletizing abnormality is determined based on the image of the depalletizing area.

[0091] Exemplary types of depalletizing anomalies include the following:

[0092] The material is still on the stack and the robot is judged to have failed to grab it;

[0093] The material is not on the stack, and the robot is judged to have dropped the material during the grasping process;

[0094] The material is on the conveyor or pallet, but the material is damaged.

[0095] It should be noted that the present embodiment does not limit the type of exceptions, and the classification of exceptions may be increased according to various possibilities that may occur in a real environment, and different processing strategies may be set based on different exception types.

[0096] Step S4: According to the type of depalletizing exception, a depalletizing exception handling option box pops up to confirm whether to perform manual handling.

[0097] In this embodiment, after determining the type of depalletizing exception, a depalletizing exception handling option box pops up. Specifically, when the material is still on the stack, that is, it has not been grabbed by the robot (grasping failure), no manual processing is required at this time, and the robot is directly controlled to perform secondary grabbing.

[0098] In one possible situation, the robot fails to grab the material and drops it. In this case, select "manual handling required" in the exception handling option box that pops up on the HMI. The human will check whether the material is damaged. If not, the material will be moved to the conveyor belt or material frame (pallet). If it is damaged, it will be handled according to the handling plan for damaged materials in the operation manual, for example: recycling, or terminating the current depalletizing order.

[0099] Step S5: If manual processing is required, an exception handling task is issued to the staff in the real environment.

[0100] In this embodiment, when manual processing is selected, a reminder is sent to the staff in the real environment, and the type of exception to be processed and the corresponding processing strategy are informed.

[0101] In this embodiment, a simulation environment is built according to the real depalletizing scene provided by the user. When an abnormality occurs in the depalletizing process in the real environment, a depalletizing abnormality reminder pops up in the simulation environment; the type of depalletizing abnormality is determined by the visual system deployed in the real environment; according to the type of depalletizing abnormality, a depalletizing abnormality handling option box pops up to confirm whether manual processing is required; if manual processing is required, an abnormality handling task is issued to the staff in the real environment. Therefore, the depalletizing process in the real environment can be monitored in real time through the virtual stack type in the simulation, so that the user can complete the remote monitoring of material disassembly in the simulation, and timely handle the stacking abnormality in the disassembly process, which has strong reliability and is convenient for tracking and managing the real stacking process.

[0102] Optionally, the method may further include: step S6, switching to a manual processing interface, and inputting the quantity of materials that need to be manually processed on the manual processing interface.

[0103] In this embodiment, if a trigger prompt message indicating that manual processing is required is received, the manual processing interface is switched to, and the quantity of materials requiring manual processing is entered on the manual processing interface.

[0104] Step S7: If manual processing is required, an exception handling task is issued to the staff in the real environment.

[0105] In this embodiment, if the robot fails to grab and the material is not on the stack, the staff is instructed to check whether the dropped material is damaged. If the material is not damaged, the staff will move the material to a conveyor belt or a pallet; the staff will receive the abnormality elimination indication information feedback from the staff, and the depalletizing task progress in the order information will be automatically increased according to the number of materials manually processed; if the material is damaged, the staff will recycle the material or terminate the current depalletizing order. In this way, the staff can be notified in time to handle the depalletizing abnormality and the progress of the abnormality handling can be monitored in a simulation environment.

[0106] Step S8, detecting whether the depalletizing task progress in the order information increases according to the preset waiting period, if so, executing step S9, if not, executing step S10.

[0107] Step S9, determine whether the depalletizing task increase progress is consistent with the number of materials that need to be manually processed input on the manual processing interface, if not, execute step S10, if yes, execute step S11.

[0108] Step S10: After entering the next waiting cycle, return to execute step S9.

[0109] Step S11, displaying a prompt message indicating that the depalletizing exception processing has been completed, receiving a start trigger instruction message sent by the user, and continuing to execute the unfinished depalletizing order.

[0110] In one possible implementation, after displaying a prompt message indicating that depalletizing exception processing has been completed, it is determined whether the depalletizing process in the simulation environment is in step with the depalletizing process in the real environment; if not, the stack type in the simulation environment is adjusted according to the stack type in the real environment so that the stack type in the simulation environment is consistent with the stack type in the real environment.

[0111] In this embodiment, by detecting whether the progress of the depalletizing task in the order information increases according to the preset waiting cycle, if the progress of the depalletizing task increases, and the increased progress is consistent with the number of materials that need to be manually processed input on the manual processing interface, a prompt message indicating that the depalletizing exception processing has been completed is displayed, and the start trigger instruction information sent by the user is received, and the unfinished depalletizing order continues to be executed; if the progress of the depalletizing task does not increase, or the increased progress of the depalletizing task is inconsistent with the number of materials that need to be manually processed input on the manual processing interface, the next waiting cycle is entered. In this way, the progress of the depalletizing order can be monitored, and it is clear whether all the exceptions that need to be manually processed have been processed, ensuring that the depalletizing process in the simulation environment is in step with the depalletizing process in the real environment, so as to simulate the depalletizing process in the real environment through the simulation environment and achieve the purpose of remote monitoring.

[0112] Fig. 9 This is a schematic diagram of the structure of a depalletizing exception handling device provided in an embodiment of the present application. The depalletizing exception handling device 900 in this embodiment may include: a processor 901 and a memory 902 .

[0113] The memory 902 is used to store programs; the memory 902 may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory 902 is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories 902. And the above computer programs, computer instructions, data, etc. can be called by the processor 901.

[0114] The above-mentioned computer programs, computer instructions, etc. may be stored in partitions in one or more memories 902 . And the above-mentioned computer programs, computer instructions, data, etc. may be called by the processor 901 .

[0115] The processor 901 is used to execute the computer program stored in the memory 902 to implement each step of the method involved in the above embodiment.

[0116] For details, please refer to the relevant description in the previous method embodiment.

[0117] The processor 901 and the memory 902 may be independent structures or integrated structures. When the processor 901 and the memory 902 are independent structures, the memory 902 and the processor 901 may be coupled and connected via a bus 903 .

[0118] The depalletizing exception handling device 900 of this embodiment can execute the technical solution in the above method. Its specific implementation process and technical principles refer to the relevant description in the above method, which will not be repeated here.

[0119] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0120] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When at least one processor of a user device executes the computer-executable instructions, the user device executes the above-mentioned various possible methods.

[0121] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0122] The present application also provides a program product, which includes a computer program. The computer program is stored in a readable storage medium. At least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server implements any method of the above-mentioned embodiments of the present invention.

[0123] Those skilled in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc. Various media that can store program codes.

[0124] Fig.10 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig.10 As shown, a program product 1000 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0125] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0126] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0127] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0128] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0129] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A depalletizing abnormality handling system, characterized in that: include: A processing unit, a display unit, an input unit, and a sending unit, wherein the processing unit is loaded with simulation software for building a simulation environment according to a real depalletizing scene provided by a user, wherein the simulation environment is used to simulate the depalletizing process in a real scene; the depalletizing process in the simulation environment is in step with the depalletizing process in the real environment; The processing unit is in communication with the robot vision system, and when the robot vision system detects that an abnormality occurs in the real environment during the depalletizing process, a depalletizing abnormality reminder pops up in the simulation interface of the display unit; The processing unit receives the type of depalletizing anomaly fed back by the robot vision system; The display unit is also used to pop up a depalletizing exception handling option box according to the type of depalletizing exception, so as to confirm whether to perform manual handling; The input unit is used to receive the first trigger operation information input by the user. If the user selects that manual processing is required, the exception handling task is issued to the staff in the real environment through the sending unit.

2. The depalletizing abnormality handling system according to claim 1, characterized in that: The robot vision system includes: an image processor, and a plurality of cameras arranged around the depalletizing area and / or on the robot; The camera transmits the collected image of the depalletizing area to the image processor; The image processor analyzes the image of the depalletizing area to determine the type of depalletizing anomaly.

3. The depalletizing abnormality handling system according to claim 2, characterized in that: The types of depalletizing anomalies include: The material is still on the stack, and the robot is judged to have failed to grab it; The material is not on the stack, and it is determined that the robot has dropped the material during the grasping process; The material is on the conveyor or pallet, but the material is damaged.

4. The depalletizing abnormality handling system according to any one of claims 1 to 3, characterized in that: The input unit is further used to receive trigger indication information for the depalletizing exception processing option box, wherein the depalletizing exception processing option box includes: depalletizing exception type, and options for whether manual processing is required; When receiving a trigger prompt message that requires manual processing, the display unit switches the current interface to the manual processing interface; The input unit is also used to receive the quantity of materials that need to be manually processed, which is input by the user on the manual processing interface.

5. The depalletizing abnormality handling system according to claim 4, characterized in that: The processing unit is specifically used to: if the robot fails to grasp and the material is not on the stack, instruct the staff to check whether the dropped material is damaged through the sending unit. If the material is not damaged, the staff will move the material to a conveyor belt or a pallet; if the material is damaged, instruct the staff to recycle the material or terminate the current depalletizing order through the sending unit; The input unit is also used to receive the abnormality elimination indication information fed back by the staff; After receiving the indication information that the abnormality has been eliminated, the processing unit increases the progress of the depalletizing task in the order information according to the number of materials manually processed.

6. The depalletizing abnormality handling system according to claim 5, characterized in that: Also includes: The depalletizing task progress detection unit is used to detect whether the depalletizing task progress in the order information increases according to a preset waiting period. If the depalletizing task progress increases, and the increased progress is consistent with the number of materials that need to be manually processed input on the manual processing interface, a prompt message indicating that the depalletizing exception processing has been completed is displayed through the display unit, and a start trigger indication message sent by the user is received through the input unit to continue to execute the unfinished depalletizing order; If the depalletizing task progress does not increase, or the increased depalletizing task progress is inconsistent with the number of materials that need to be manually processed entered on the manual processing interface, the next waiting cycle is entered.

7. The depalletizing abnormality handling system according to claim 6, characterized in that: Also includes: A depalletizing step confirmation unit, used to determine whether the depalletizing process in the simulation environment is consistent with the depalletizing process in the real environment after displaying a prompt message that the depalletizing exception processing has been completed; If they are inconsistent, the processing unit adjusts the stacking type in the simulation environment according to the stacking type in the real environment, so that the stacking type in the simulation environment is consistent with the stacking type in the real environment.

8. The depalletizing abnormality handling system according to claim 6, characterized in that: Also includes: The waiting timeout prompt unit is used to display a waiting timeout prompt box through the display unit when the number of waiting cycles reaches the upper limit threshold, so as to remind the staff to check the cause of the timeout.

9. The depalletizing abnormality handling system according to any one of claims 1 to 3, characterized in that: Also includes: The storage unit is used to record the time spent on manually handling depalletizing exceptions, the number of times the exceptions are handled, and the information of the staff who handles each exception.

10. A robot, characterized in that: The robot comprises a robot body, a drive system and a robotic arm, wherein the robot is used to perform depalletizing operations according to preset depalletizing rules, and a processor in the robot body, or a computer remotely connected to the robot body, is loaded with a depalletizing exception handling system as claimed in any one of claims 1 to 9, and is used to pop up a depalletizing exception reminder in a simulation environment when an exception occurs during the depalletizing process in a real environment; Determine the type of depalletizing anomaly through a visual system deployed in a real environment; According to the type of depalletizing exception, a depalletizing exception handling option box pops up to confirm whether to perform manual processing; If manual processing is required, issue exception handling tasks to staff in the real environment.

Images