Collaborative methods and systems for industrial robots under multi-sensory fusion

Abstract

The present invention discloses a collaborative method and system under multi-sensory fusion of an industrial robot, which outputs collaborative instructions of multiple mechanical arms according to environmental results and acquired action instructions; in the process of executing the collaborative instructions, the multiple mechanical arms are dynamically regulated during the working process and adjusted according to the environmental parameters collected in real time; keywords are determined based on the collaborative instructions, and a clamping model is constructed according to the keywords and the multiple mechanical arms; in the clamping model, multiple clamping points are defined based on an object to be clamped, and fixed-point clamping of multiple mechanical arms is triggered according to the multiple clamping points, and dynamic balance is performed based on the clamping forces applied by the multiple mechanical arms. At this time, the multiple mechanical arms perform collaborative operations according to the collaborative instructions, and environmental parameters are introduced to facilitate environmental control of the multiple mechanical arms during operation, thereby facilitating dynamic adjustment of the multiple mechanical arms during the collaborative process to overcome changes in environmental parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial robot collaboration under multi-sensory fusion, and in particular to an industrial robot collaboration method and system under multi-sensory fusion. Background Art

[0002] With the development of science and technology, industrial robots are applied to industrial scenes and use multiple robotic arms to perform actions. At this time, the industrial robot includes a machine body and multiple robotic arms. The multiple robotic arms are connected to the machine body and distributed at different positions of the machine body. In the existing technology, each robotic arm is responsible for the corresponding operation, and a single robotic arm is used for real-time operation. The coordinated action between multiple robotic arms is only superimposed according to the preset action, and cannot adapt to the surrounding environment and dynamic adjustment. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a collaborative method and system under multi-sensory fusion of an industrial robot. Multiple robotic arms perform collaborative operations according to collaborative instructions, and introduce environmental parameters to facilitate environmental control of the multiple robotic arms during operation, thereby facilitating dynamic adjustment of the multiple robotic arms during the collaborative process to overcome changes in environmental parameters. At the same time, keywords are determined based on the collaborative instructions, and a clamping model is constructed based on the keywords and multiple robotic arms. The dynamic balance of multiple robotic arms is constructed in the clamping model to ensure stable clamping of the object to be clamped during movement.

[0004] In order to solve the above technical problems, an embodiment of the present invention provides a collaborative method of an industrial robot under multi-sensory fusion, which is applied to a collaborative scenario of an industrial robot under multi-sensory fusion; the collaborative method of an industrial robot under multi-sensory fusion includes:

[0005] Collect data from multiple robotic arms in industrial robots and associate multiple robotic arms;

[0006] Each robot arm is equipped with an environmental sensing component, which senses environmental parameters in different directions;

[0007] The plurality of environmental parameters are formed into an environmental parameter set, and the environmental result is matched based on the environmental parameter set;

[0008] Output coordination instructions of multiple robot arms according to the environmental results and the obtained action instructions;

[0009] In the process of executing collaborative instructions, multiple robotic arms are dynamically controlled during the work process and adjusted according to the environmental parameters collected in real time;

[0010] Determine keywords based on collaborative instructions, and build a gripping model based on the keywords and multiple robotic arms;

[0011] In the clamping model, multiple clamping points are defined based on the object to be clamped, and fixed-point clamping of multiple robotic arms is triggered according to the multiple clamping points, and dynamic balance is performed based on the clamping forces applied by the multiple robotic arms.

[0012] Optionally, the collecting of multiple mechanical arms in the industrial robot and associating the multiple mechanical arms includes:

[0013] Locate industrial robots and respond to them in all directions;

[0014] Obtain response results, and determine the positions and operating ranges of multiple robotic arms in the response results;

[0015] Collect the positions of multiple robotic arms;

[0016] Defining relative positions of the plurality of robotic arms based on the positions of the plurality of robotic arms, and constructing an operation space according to the relative positions and the operation range;

[0017] Multiple robotic arms are associated, and the degree of coordination of the multiple robotic arms in the associated state is trained. At this time, the operation points are arranged in sequence along the circular direction in the operation space, and the coordination relationship of the multiple robotic arms is triggered based on the operation points.

[0018] Optionally, each robot arm is equipped with an environmental sensing component, and the environmental parameters in different directions are sensed by the environmental sensing component, including:

[0019] Each robot arm is equipped with an environment sensing component. At this time, the environment sensing component is located on one side of the operating end of the robot arm and is arranged at a distance from the operating end;

[0020] Collect the current direction of the operation terminal;

[0021] Adjust the orientation of the sensing end of the environment sensing component based on the current orientation of the operating end, and collect the environment images output by each environment sensing component on the same robotic arm;

[0022] forming a perception image based on the synthesis of the environmental images, and marking the position of the operating end in the perception image;

[0023] If the position of the operating end is not at the center of the perceived image, the offset between the position of the operating end and the center is collected, and each environmental sensing element is adjusted according to the offset until the position of the operating end is at the center of the perceived image;

[0024] The environment sensing element senses environmental parameters in different directions and marks the environmental parameters on the outer peripheral side of the operating end.

[0025] Optionally, forming an environmental parameter set from the multiple environmental parameters and matching the environmental result based on the environmental parameter set includes:

[0026] Collect multiple environmental parameters and mark corresponding positions of multiple environmental parameters;

[0027] The plurality of environmental parameters are formed into an environmental parameter set, and a perception region is constructed based on the plurality of locations, wherein the environmental parameter set is a part of the perception region;

[0028] Dividing the perception area into multiple sub-perception areas, each sub-perception area contains at least two different types of environmental parameters;

[0029] In the sub-sensing area, determining an environmental event based on a plurality of environmental parameters in the sub-sensing area;

[0030] Combine the environmental events connected to each sub-perception area and match the environmental results of the perception area.

[0031] Optionally, the step of outputting coordination instructions of multiple robot arms according to the environmental results and the acquired action instructions includes:

[0032] Collect environmental results and obtained action instructions;

[0033] Associating the environmental results with the obtained action instructions, and collecting adaptation parameters of the environmental results and the obtained action instructions;

[0034] Defining the adaptation level according to the adaptation parameters;

[0035] If the adaptation level is lower than the preset level threshold, the modification of the action instruction is triggered, and the action instruction is partially modified according to the environmental result to form a modified action instruction;

[0036] Divide the modified action instruction into a plurality of sub-action instructions;

[0037] Match the coordination instructions of multiple robot arms according to multiple sub-action instructions.

[0038] Optionally, in the process of executing the collaborative instruction, the multiple robotic arms are dynamically regulated during the working process and adjusted according to the environmental parameters collected in real time, including:

[0039] In the process of executing the collaborative instruction, multiple robotic arms move along the corresponding sub-action instructions;

[0040] Monitor the movements of multiple robotic arms in real time and collect the corresponding movement routes.

[0041] Optionally, in the process of executing the collaborative instruction, the multiple robotic arms are dynamically regulated during the working process and adjusted according to the environmental parameters collected in real time, further comprising:

[0042] Perform environmental perception along the action route and define the surrounding environment of the action route;

[0043] The activity of the robot arm in the action route is defined according to the surrounding environment of the action route, and the action of the robot arm is adjusted according to the environmental parameters collected in real time.

[0044] Optionally, determining keywords based on the collaborative instruction and constructing a gripping model according to the keywords and multiple robotic arms includes:

[0045] Collect collaborative instructions;

[0046] Determine keywords based on collaborative instructions and define the action types of keywords;

[0047] Associate the corresponding robot arm according to the action type of the keyword;

[0048] If the action type of the keyword is a gripping type, a gripping model is constructed based on multiple robotic arms;

[0049] In the clamping model, construct the active and passive relationships of multiple robotic arms;

[0050] The coordination of multiple robotic arms in the clamping process is triggered according to the active relationship and the slave relationship, and the coordination efficiency between the multiple robotic arms is detected in real time. At the same time, the coordination efficiency between the multiple robotic arms is regulated according to the response logic between the multiple robotic arms.

[0051] Optionally, in the clamping model, multiple clamping points are defined based on the object to be clamped, and fixed-point clamping of multiple robotic arms is triggered according to the multiple clamping points, and dynamic balance is performed based on the clamping forces applied by the multiple robotic arms, including:

[0052] In the clamping model, the spatial position of the object to be clamped is collected;

[0053] Perform multi-directional shooting based on the spatial position of the object to be clamped, and output multiple images of the object to be clamped in different directions;

[0054] A three-dimensional model of the object to be clamped is constructed based on the multiple images, and multiple clamping points are defined according to the three-dimensional model of the object to be clamped. At this time, the three-dimensional model of the object to be clamped performs a dynamic drop simulation in the virtual scene and outputs multiple clamping points;

[0055] Triggering fixed-point clamping of multiple robotic arms according to multiple clamping points, at this time, matching the corresponding robotic arms based on the position and orientation of the clamping points;

[0056] Collect the clamping force applied by each clamping point, and construct a force distribution map of the object to be clamped based on multiple clamping forces;

[0057] When the object to be clamped moves with the movement of the industrial robot, the change parameters in the force distribution diagram are monitored, and the dynamic adjustment of the corresponding robot arm is triggered based on the change amount of the change parameters to ensure the dynamic balance of the object to be clamped during the movement.

[0058] Optionally, a collaborative system under multi-sensory fusion of an industrial robot is characterized in that the collaborative system under multi-sensory fusion of an industrial robot is applied to the collaborative method under multi-sensory fusion of an industrial robot, and the collaborative system under multi-sensory fusion of an industrial robot includes:

[0059] A collection module, used to collect data from multiple robotic arms in an industrial robot and associate the multiple robotic arms;

[0060] Environmental parameter module, used to equip each robot arm with an environmental sensing component, and sense environmental parameters in different directions through the environmental sensing component;

[0061] An environment result module, used to form an environment parameter set from multiple environment parameters, and match environment results based on the environment parameter set;

[0062] A coordination instruction module is used to output coordination instructions of multiple robot arms according to environmental results and acquired motion instructions;

[0063] The dynamic control module is used to dynamically control the working process of multiple robotic arms during the execution of collaborative instructions, and adjust them according to the environmental parameters collected in real time;

[0064] A gripping module, used to determine keywords based on collaborative instructions and build a gripping model based on the keywords and multiple robotic arms;

[0065] The dynamic balancing module is used to define multiple clamping points based on the object to be clamped in the clamping model, trigger the fixed-point clamping of multiple robotic arms according to the multiple clamping points, and perform dynamic balancing based on the clamping forces applied by the multiple robotic arms.

[0066] In an embodiment of the present invention, through the method in the embodiment of the present invention, multiple robotic arms in an industrial robot are collected and associated; each robotic arm is equipped with an environmental sensing component, and the environmental parameters in different directions are sensed by the environmental sensing component; the multiple environmental parameters are formed into an environmental parameter set, and the environmental results are matched based on the environmental parameter set; the coordination instructions of the multiple robotic arms are output according to the environmental results and the acquired action instructions; in the process of executing the coordination instructions, the multiple robotic arms are dynamically regulated during the working process and adjusted according to the environmental parameters collected in real time; keywords are determined based on the coordination instructions, and a clamping model is constructed according to the keywords and the multiple robotic arms. ; In the clamping model, multiple clamping points are defined based on the object to be clamped, and fixed-point clamping of multiple robotic arms is triggered according to the multiple clamping points, and dynamic balance is performed based on the clamping force applied by the multiple robotic arms. At this time, the multiple robotic arms perform collaborative operations according to collaborative instructions, and environmental parameters are introduced to facilitate environmental control of the multiple robotic arms during operation, thereby facilitating dynamic adjustment of the multiple robotic arms during the collaborative process to overcome changes in environmental parameters. At the same time, keywords are determined based on collaborative instructions, and a clamping model is constructed based on the keywords and multiple robotic arms. A dynamic balance of multiple robotic arms is constructed in the clamping model to ensure stable clamping of the object to be clamped during movement. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 is a flowchart of a collaborative method for an industrial robot under multi-sensory fusion in an embodiment of the present invention;

[0069] Figure 2 is a flow chart of S11 in the collaborative method under multi-sensory fusion of an industrial robot in an embodiment of the present invention;

[0070] Figure 3 is a flow chart of S12 in the collaborative method under multi-sensory fusion of an industrial robot in an embodiment of the present invention;

[0071] Figure 4 is a flow chart of S13 in the collaborative method under multi-sensory fusion of an industrial robot in an embodiment of the present invention;

[0072] Figure 5 is a flow chart of S14 in the collaborative method under multi-sensory fusion of an industrial robot in an embodiment of the present invention;

[0073] Figure 6 is a flow chart of S15 in the collaborative method under multi-sensory fusion of an industrial robot in an embodiment of the present invention;

[0074] Figure 7 is a flow chart of S16 in the collaborative method under multi-sensory fusion of an industrial robot in an embodiment of the present invention;

[0075] Figure 8 Schematic diagram of the structure of the collaborative system of the industrial robot under multi-sensory fusion in an embodiment of the present invention;

[0076] Fig. 9 The figure is a hardware diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Example

[0079] See also Figures 1 to 9 , a collaborative method for industrial robots under multi-sensory fusion, applied to collaborative scenarios under multi-sensory fusion of industrial robots; the collaborative method for industrial robots under multi-sensory fusion includes:

[0080] Step S11: collecting multiple mechanical arms in the industrial robot and associating the multiple mechanical arms;

[0081] Step S12: Each robot arm is equipped with an environmental sensing component, and the environmental parameters in different directions are sensed by the environmental sensing component;

[0082] Step S13: forming an environmental parameter set from multiple environmental parameters, and matching environmental results based on the environmental parameter set;

[0083] Step S14: outputting coordination instructions of multiple robot arms according to the environmental results and the acquired action instructions;

[0084] Step S15: During the execution of the collaborative instruction, the multiple robotic arms are dynamically regulated during the work process and adjusted according to the environmental parameters collected in real time;

[0085] Step S16: determining keywords based on the collaborative instructions, and constructing a gripping model based on the keywords and multiple robotic arms;

[0086] Step S17: In the clamping model, multiple clamping points are defined based on the object to be clamped, and fixed-point clamping of multiple robotic arms is triggered according to the multiple clamping points, and dynamic balance is performed based on the clamping forces applied by the multiple robotic arms.

[0087] In an embodiment of the present invention, through the method in the embodiment of the present invention, multiple robotic arms perform collaborative operations according to collaborative instructions, and environmental parameters are introduced to facilitate environmental control of the multiple robotic arms during operation, thereby facilitating dynamic adjustment of the multiple robotic arms during the collaborative process to overcome changes in environmental parameters. At the same time, keywords are determined based on the collaborative instructions, and a clamping model is constructed based on the keywords and multiple robotic arms. The dynamic balance of multiple robotic arms is constructed in the clamping model, which ensures stable clamping of the object to be clamped during movement.

[0088] In step S11, multiple mechanical arms in the industrial robot are collected and the multiple mechanical arms are associated;

[0089] In the specific implementation process of the present invention, the specific steps may be:

[0090] S111: Locate industrial robots and respond to them in all directions;

[0091] S112: Obtain response results, and determine the positions and operating ranges of multiple robot arms in the response results;

[0092] S113: collecting positions of multiple robotic arms;

[0093] S114: defining relative positions of the plurality of robotic arms based on the positions of the plurality of robotic arms, and constructing an operation space according to the relative positions and the operation range;

[0094] S115: Associating multiple robotic arms and training the coordination degree of the multiple robotic arms in the associated state. At this time, the operation points are arranged in sequence along the circular direction in the operation space, and the coordination relationship of the multiple robotic arms is triggered based on the operation points.

[0095] In an embodiment of the present application, an industrial robot is positioned and an all-round response is performed on the industrial robot, so that the positioning of multiple robotic arms can be triggered by the all-round response, thereby determining the positions and operating ranges of the multiple robotic arms in the response results, so as to facilitate self-inspection of the robotic arms of the industrial robot.

[0096] At this time, the positions of multiple robotic arms are collected, the relative positions of the multiple robotic arms are defined based on the positions of the multiple robotic arms, and the operation space is constructed according to the relative positions and the operation range; the multiple robotic arms are associated, and the degree of coordination of the multiple robotic arms in the associated state is trained. At this time, the operation points are arranged in sequence along the circular direction in the operation space, and the coordination relationship of the multiple robotic arms is triggered based on the operation points, so as to clarify the coordination relationship between the multiple robotic arms and ensure the coordination between the multiple robotic arms.

[0097] If there is no collaborative relationship between multiple robotic arms, collaborative training is performed on the multiple robotic arms, and multiple operation points are used to train the multiple robotic arms, so that multiple robotic arms can be triggered to run simultaneously at one operation point, so that the collaborative relationship between the multiple robotic arms can be loaded during operation, thereby realizing complementary actions between the multiple robotic arms.

[0098] In step S12, each robot arm is equipped with an environmental sensing component, and the environmental parameters in different directions are sensed by the environmental sensing component;

[0099] In the specific implementation process of the present invention, the specific steps may be:

[0100] S121: Each robot arm is equipped with an environment sensing component. At this time, the environment sensing component is located on one side of the operating end of the robot arm and is spaced apart from the operating end;

[0101] S122: collecting the current direction of the operation terminal;

[0102] S123: adjusting the orientation of the sensing end of the environment sensing component based on the current orientation of the operating end, and collecting environment images output by each environment sensing component on the same robotic arm;

[0103] S124: forming a perception image based on the synthesis of the environmental images, and marking the position of the operating end in the perception image;

[0104] S125: If the position of the operating end is not at the center of the perceived image, an offset between the position of the operating end and the center is collected, and each environmental sensing element is adjusted according to the offset until the position of the operating end is at the center of the perceived image;

[0105] S126: The environmental sensing element senses environmental parameters in different directions and marks the environmental parameters on the outer peripheral side of the operating end.

[0106] In an embodiment of the present application, a robotic arm is equipped with an environmental sensing component, which is located on one side of the operating end of the robotic arm and is spaced apart from the operating end so that the surrounding environment of the robotic arm can be sensed by the environmental sensing component, thereby collecting environmental parameters of the surrounding environment of the robotic arm. At the same time, the orientation of the sensing end of the environmental sensing component is regulated based on the current direction of the operating end, and environmental images output by each environmental sensing component on the same robotic arm are collected.

[0107] For environmental images in different directions, multiple environmental images are synthesized to form a perceived image, thereby performing overall control of the perceived image. At this time, the position of the operating end in the perceived image is marked. If the position of the operating end is not at the center of the perceived image, the offset between the position of the operating end and the center is collected, and each environmental sensing component is adjusted according to the offset until the position of the operating end is at the center of the perceived image, thereby ensuring that the position of the operating end is at the center of the perceived image, and then the peripheral environment of the operating end can also be guaranteed. In addition, the environmental sensing component senses environmental parameters in different directions and marks the environmental parameters on the peripheral side of the operating end, thereby performing real-time environmental control of the operating end.

[0108] If the operating end is about to approach an obstacle during the operation, the obstacle avoidance operation is triggered based on the corresponding robotic arm. At this time, the position of the obstacle is marked in the environmental image, the position of the obstacle and the position of the operating end are located, and the distance between the position of the obstacle and the position of the operating end is detected in real time. If the distance between the position of the obstacle and the position of the operating end is lower than the preset distance threshold, the robotic arm is adjusted and moved in the opposite direction of the obstacle, so that the operating end can urgently move away from the obstacle, thereby ensuring the normal operation of the robotic arm.

[0109] In step S13, multiple environmental parameters are formed into an environmental parameter set, and environmental results are matched based on the environmental parameter set;

[0110] In the specific implementation process of the present invention, the specific steps may be:

[0111] S131: Collect multiple environmental parameters, and mark corresponding positions of the multiple environmental parameters;

[0112] S132: forming an environmental parameter set from the multiple environmental parameters, and constructing a perception area based on the multiple locations, with the environmental parameter set serving as a part of the perception area;

[0113] S133: Divide the perception area into multiple sub-perception areas, each sub-perception area containing at least two different types of environmental parameters;

[0114] S134: determining, in the sub-sensing area, an environmental event based on a plurality of environmental parameters in the sub-sensing area;

[0115] S135: Combine the environmental events connected to each sub-perception area and match the environmental results of the perception area.

[0116] In an embodiment of the present application, multiple environmental parameters are collected, the positions corresponding to the multiple environmental parameter marks are grouped to form an environmental parameter set, so as to control the environmental parameter set as a whole. At the same time, a perception area is constructed based on the multiple positions, and the environmental parameter set serves as a part of the perception area.

[0117] Furthermore, the perception area is divided into multiple sub-perception areas, each of which contains at least two different types of environmental parameters. In the sub-perception areas, environmental events are determined based on multiple environmental parameters in the sub-perception areas to facilitate the collection of multiple environmental events. At this time, the environmental events connected to each sub-perception area are combined, and the environmental results of the perception area are matched, so that the environmental results of the perception area are known based on the combination of multiple environmental events, thereby achieving in-depth control of the environmental results of the perception area and ensuring the accuracy of the environmental results of the perception area.

[0118] In another embodiment of the present application, a multi-sensor fusion algorithm is provided based on the fusion of laser point cloud features and visual H channel features. The laser point cloud feature extraction algorithm is analyzed and tested, and the point cloud data in the laser radar acquisition environment is filtered, clustered, and other processing is performed to extract the features of the ground, obstacles, etc., and a suitable feature extraction algorithm is selected in combination with the specific scene. The visual image feature extraction algorithm is studied, the RGB image color space is converted into an HSV image, the H channel information is extracted, and then the H channel image is subjected to edge detection, morphological processing, and other operations to extract the contours and feature points of the target object in the image. The extracted laser point cloud features and image features are input into the neural network for positioning and environmental perception, including detection and recognition of static and dynamic obstacles and people, to improve the accuracy and reliability of environmental perception.

[0119] At the same time, in the dynamic obstacle avoidance process, in order to achieve safe and flexible obstacle avoidance, different obstacle avoidance strategies are set according to the different types (pedestrians, other AMRs and other obstacles) and different states (moving, stationary) of obstacles that appear on the planned path: people-warning avoidance, mobile robots-traffic rules, other obstacles-dynamic obstacles slow down in advance, and static obstacles combined obstacle avoidance strategy. In the dynamic obstacle avoidance process, different obstacle avoidance algorithms are adopted for obstacles in different states according to the perceived obstacle type and its obstacle avoidance strategy. For stationary obstacles, an improved rolling window method is used. The global path planning path is used as the initial value. The sensors carried by the mobile robot are used to perceive the local environment information in real time. In each local planning window, the minimum distance to the end point is used as the benchmark or the minimum angle between the robot's expected movement direction and the direction of the robot's current position pointing to the target point is used as the benchmark to ensure that each local planning path is optimal or suboptimal, so that the robot avoids obstacles and moves toward the target point; for moving obstacles, a deep reinforcement learning method is studied, radar perception, visual perception and other information are used as input, and a reward function is set. Different obstacle avoidance strategies are adopted according to different obstacle categories to achieve safe and flexible obstacle avoidance. Cross-experiments are conducted to obtain the optimal obstacle avoidance solution.

[0120] S14: Outputting coordination instructions of multiple robot arms according to the environmental results and the acquired action instructions;

[0121] In the specific implementation process of the present invention, the specific steps may be:

[0122] S141: Collecting environmental results and acquired action instructions;

[0123] S142: Associating the environment result with the obtained action instruction, and collecting adaptation parameters of the environment result and the obtained action instruction;

[0124] S143: defining an adaptation level according to the adaptation parameters;

[0125] S144: If the adaptation level is lower than the preset level threshold, the modification of the action instruction is triggered, and the action instruction is partially modified according to the environmental result to form a modified action instruction;

[0126] S145: dividing the modified action instruction into a plurality of sub-action instructions;

[0127] S146: Matching the coordination instructions of multiple robot arms according to the multiple sub-action instructions.

[0128] In an embodiment of the present application, environmental results and obtained action instructions are introduced to facilitate associating the environmental results and the obtained action instructions, thereby collecting adaptation parameters of the environmental results and the obtained action instructions, and reflecting the adaptability of the environmental results and the obtained action instructions through the adaptation parameters.

[0129] At this time, the adaptation level is defined according to the adaptation parameters; if the adaptation level is lower than the preset level threshold, the modification of the action instruction is triggered, and the action instruction is partially modified according to the environmental result to form a modified action instruction; based on the modified action instruction, it is divided into multiple sub-action instructions; according to the multiple sub-action instructions, the collaborative instructions of multiple robotic arms are matched. At this time, the adaptation level is introduced to ensure the adaptability between the environmental results and the obtained action instructions, and the action instructions are partially modified based on the environmental results to facilitate the adjustment of the collaborative instructions, thereby realizing the corresponding regulation of the collaborative instructions with the environmental results.

[0130] S15: In the process of executing the collaborative instructions, multiple robotic arms are dynamically controlled during the work process and adjusted according to the environmental parameters collected in real time;

[0131] In the specific implementation process of the present invention, the specific steps may be:

[0132] S151: In the process of executing the coordinated instruction, the multiple robot arms act according to the corresponding sub-action instructions;

[0133] S152: monitor the movements of multiple robotic arms in real time and collect corresponding movement routes;

[0134] S153: Perform environmental perception along the action route and define the surrounding environment of the action route;

[0135] S154: defining the amount of movement of the robot arm in the movement route according to the surrounding environment of the movement route, and adjusting the movement of the robot arm according to the environmental parameters collected in real time.

[0136] In an embodiment of the present application, in the process of executing collaborative instructions, multiple robotic arms are dynamically regulated during the work process and adjusted according to the environmental parameters collected in real time, so that the multiple robotic arms can make adaptive adjustments based on the environment. At the same time, the movements of the multiple robotic arms are monitored in real time, and the corresponding movement routes are collected to perform environmental perception based on the movement routes.

[0137] At this time, environmental perception is performed along the action route, and the surrounding environment of the action route is defined; the activity of the robot arm on the action route is defined according to the surrounding environment of the action route. At the same time, the movement of the robot arm is adjusted according to the environmental parameters collected in real time, so as to ensure timely adjustment of the movement of the robot arm.

[0138] S16: determining keywords based on the collaborative instructions, and building a gripping model based on the keywords and multiple robotic arms;

[0139] In the specific implementation process of the present invention, the specific steps may be:

[0140] S161: Collecting collaborative instructions;

[0141] S162: Determine a keyword according to the collaborative instruction, and define an action type of the keyword;

[0142] S163: Associating the corresponding robot arm according to the action type of the keyword;

[0143] S164: If the action type of the keyword is a clamping type, a clamping model is constructed based on multiple robotic arms;

[0144] S165: In the clamping model, construct the active and passive relationships of multiple robotic arms;

[0145] S166: Trigger the coordination of multiple robotic arms in the clamping process according to the active relationship and the passive relationship, and detect the coordination efficiency between the multiple robotic arms in real time. At the same time, adjust the coordination efficiency between the multiple robotic arms according to the response logic between the multiple robotic arms.

[0146] In an embodiment of the present application, multiple robotic arms perform collaborative operations according to collaborative instructions, and environmental parameters are introduced to facilitate environmental control of the multiple robotic arms during operation, thereby facilitating dynamic adjustment of the multiple robotic arms during the collaborative process to overcome changes in environmental parameters.

[0147] At this time, the keyword is determined according to the collaborative instruction, and the action type of the keyword is defined; the corresponding robot arm is associated according to the action type of the keyword; if the action type of the keyword is a clamping type, a clamping model is constructed based on multiple robot arms. Therefore, the collaborative relationship of multiple robot arms is fully utilized, and the corresponding model is constructed based on the keyword of the collaborative instruction, thereby ensuring the completion of the clamping model.

[0148] Furthermore, in the clamping model, the active relationship and the driven relationship of multiple robotic arms are constructed; the coordination of multiple robotic arms in the clamping process is triggered according to the active relationship and the driven relationship, and the coordination efficiency between the multiple robotic arms is detected in real time. At the same time, the coordination efficiency between the multiple robotic arms is regulated according to the response logic between the multiple robotic arms, thereby realizing the control of the coordination efficiency between the multiple robotic arms and ensuring the smoothness of the movements between the multiple robotic arms.

[0149] S17: In the clamping model, a plurality of clamping points are defined based on the object to be clamped, and fixed-point clamping of a plurality of robotic arms is triggered according to the plurality of clamping points, and dynamic balance is performed based on the clamping forces applied by the plurality of robotic arms;

[0150] In the specific implementation process of the present invention, the specific steps may be:

[0151] In the clamping model, the spatial position of the object to be clamped is collected; based on the spatial position of the object to be clamped, multi-directional shooting is performed, and multiple images of the object to be clamped in different directions are output; a three-dimensional model of the object to be clamped is constructed based on the multiple images, and multiple clamping points are defined according to the three-dimensional model of the object to be clamped. At this time, the three-dimensional model of the clamped object is dynamically dropped in the virtual scene and multiple clamping points are output;

[0152] Fixed-point clamping of multiple robotic arms is triggered according to multiple clamping points. At this time, the corresponding robotic arms are matched based on the position and orientation of the clamping points; the clamping force applied by each clamping point is collected, and a force distribution map of the object to be clamped is constructed based on the multiple clamping forces; when the object to be clamped moves with the movement of the industrial robot, the changing parameters in the force distribution map are monitored, and the dynamic adjustment of the corresponding robotic arms is triggered based on the change amount of the changing parameters to ensure the dynamic balance of the object to be clamped during the movement.

[0153] In addition, in a multi-machine collaborative system, multiple AMRs need to work together to avoid collisions and conflicts and maintain orderly movement to efficiently complete their respective tasks. However, such task collaboration involves a large-scale robot group, and the working environment is often dynamically changing. Obstacles, new tasks or other robot failures may occur. Therefore, a "traffic-like" multi-machine hierarchical collaborative planning framework is constructed to achieve multi-machine collaborative and efficient path planning. The main control unit is responsible for the initial path planning of multiple machines, path update information monitoring, and exception handling (path replanning and conflict adjustment). After receiving the initial path, the autonomous mobile robot autonomously avoids obstacles according to the actual scene and sends path change information to the main control unit; if there are abnormalities such as congestion, it sends a replanning or negotiation request. Collaborative global path planning is to perform multi-machine multi-objective path planning through the main control unit, adopt an intelligent optimization algorithm, study the main target method (total completion time is the main target, smoothness is the secondary target or constraint condition) multi-machine multi-objective path optimization algorithm, and find the global optimization path.

[0154] In an embodiment of the present invention, through the method in the embodiment of the present invention, multiple robotic arms in an industrial robot are collected and associated; each robotic arm is equipped with an environmental sensing component, and the environmental parameters in different directions are sensed by the environmental sensing component; the multiple environmental parameters are formed into an environmental parameter set, and the environmental results are matched based on the environmental parameter set; the coordination instructions of the multiple robotic arms are output according to the environmental results and the acquired action instructions; in the process of executing the coordination instructions, the multiple robotic arms are dynamically regulated during the working process and adjusted according to the environmental parameters collected in real time; keywords are determined based on the coordination instructions, and a clamping model is constructed according to the keywords and the multiple robotic arms. ; In the clamping model, multiple clamping points are defined based on the object to be clamped, and fixed-point clamping of multiple robotic arms is triggered according to the multiple clamping points, and dynamic balance is performed based on the clamping force applied by the multiple robotic arms. At this time, the multiple robotic arms perform collaborative operations according to collaborative instructions, and environmental parameters are introduced to facilitate environmental control of the multiple robotic arms during operation, thereby facilitating dynamic adjustment of the multiple robotic arms during the collaborative process to overcome changes in environmental parameters. At the same time, keywords are determined based on collaborative instructions, and a clamping model is constructed based on the keywords and multiple robotic arms. A dynamic balance of multiple robotic arms is constructed in the clamping model to ensure stable clamping of the object to be clamped during movement.

[0155] Example

[0156] See also Figure 8 , Figure 8 It is a schematic diagram of the structural composition of the collaborative system of the industrial robot under multi-sensory fusion in an embodiment of the present invention.

[0157] like Figure 8 As shown, a collaborative system of an industrial robot under multi-sensory fusion, the collaborative system of an industrial robot under multi-sensory fusion includes:

[0158] The acquisition module 21 is used to acquire data from multiple mechanical arms in the industrial robot and associate the multiple mechanical arms;

[0159] An environmental parameter module 22 is used to equip each robotic arm with an environmental sensing component and sense environmental parameters in different directions through the environmental sensing component;

[0160] An environment result module 23, used to form an environment parameter set from multiple environment parameters, and match environment results based on the environment parameter set;

[0161] A coordination instruction module 24, used for outputting coordination instructions of multiple robot arms according to the environmental results and the acquired action instructions;

[0162] The dynamic control module 25 is used to dynamically control the working process of multiple robot arms during the execution of the coordinated instruction, and adjust according to the environmental parameters collected in real time;

[0163] A gripping module 26, for determining keywords based on the collaborative instructions, and constructing a gripping model based on the keywords and the plurality of robotic arms;

[0164] The dynamic balancing module 27 is used to define multiple clamping points based on the object to be clamped in the clamping model, trigger the fixed-point clamping of multiple robotic arms according to the multiple clamping points, and perform dynamic balancing based on the clamping forces applied by the multiple robotic arms.

[0165] Example

[0166] See also Fig. 9 , refer to the following Fig. 9 The electronic device 40 according to this embodiment of the present invention will be described. Fig. 9 The electronic device 40 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0167] like Fig. 9 As shown, the electronic device 40 is in the form of a general computing device. The components of the electronic device 40 may include but are not limited to: at least one processing unit 41, at least one storage unit 42, and a bus 43 connecting different system components (including the storage unit 42 and the processing unit 41).

[0168] The storage unit stores program codes, which can be executed by the processing unit 41, so that the processing unit 41 executes the steps according to various exemplary embodiments of the present invention described in the above “Embodiment Method” section of this specification.

[0169] The storage unit 42 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit 421 and / or a cache storage unit 422 , and may further include a read-only storage unit (ROM) 423 .

[0170] The storage unit 42 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0171] Bus 43 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0172] The electronic device 40 may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 40, and / or any device that enables the electronic device 40 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 44. Furthermore, the electronic device 40 may also communicate with one or more networks (e.g., local area networks, wide area networks, and / or public networks, such as the Internet) via a network adapter 45. Fig. 9 As shown, the network adapter 45 communicates with other modules of the electronic device 40 via the bus 43. It should be understood that although Fig. 9 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, tape drives, and data backup planning systems.

[0173] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0174] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: a read-only memory, a random access memory, a disk or an optical disk, etc. In addition, it stores computer program instructions, and when the computer program instructions are executed by a computer, the computer executes the above method.

[0175] In addition, the collaborative method and system under multi-sensory fusion of industrial robots provided in the embodiments of the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A collaborative method for industrial robots under multi-sensory fusion, characterized in that: Applied to collaborative scenarios under multi-sensory fusion of industrial robots; The collaborative method of the industrial robot under multi-sensory fusion includes: Collect data from multiple robotic arms in industrial robots and associate multiple robotic arms; Each robot arm is equipped with an environmental sensing component, which senses environmental parameters in different directions; The plurality of environmental parameters are formed into an environmental parameter set, and the environmental result is matched based on the environmental parameter set; Output coordination instructions of multiple robot arms according to the environmental results and the obtained action instructions; In the process of executing collaborative instructions, multiple robotic arms are dynamically controlled during the work process and adjusted according to the environmental parameters collected in real time; Determine keywords based on collaborative instructions, and build a gripping model based on the keywords and multiple robotic arms; In the clamping model, multiple clamping points are defined based on the object to be clamped, and fixed-point clamping of multiple robotic arms is triggered according to the multiple clamping points, and dynamic balance is performed based on the clamping forces applied by the multiple robotic arms; The forming of an environmental parameter set from the plurality of environmental parameters and matching the environmental result based on the environmental parameter set includes: Collect multiple environmental parameters and mark corresponding positions of multiple environmental parameters; The plurality of environmental parameters are formed into an environmental parameter set, and a perception region is constructed based on the plurality of locations, wherein the environmental parameter set is a part of the perception region; Dividing the perception area into multiple sub-perception areas, each sub-perception area contains at least two different types of environmental parameters; In the sub-sensing area, determining an environmental event based on a plurality of environmental parameters in the sub-sensing area; Combine the environmental events connected to each sub-perception area and match the environmental results of the perception area; The step of outputting the coordination instructions of the multiple robot arms according to the environmental results and the acquired action instructions includes: Collect environmental results and obtained action instructions; Associating the environmental results with the obtained action instructions, and collecting adaptation parameters of the environmental results and the obtained action instructions; Defining the adaptation level according to the adaptation parameters; If the adaptation level is lower than the preset level threshold, the modification of the action instruction is triggered, and the action instruction is partially modified according to the environmental result to form a modified action instruction; Divide the modified action instruction into a plurality of sub-action instructions; Matching coordination instructions of multiple robot arms according to multiple sub-action instructions; In the clamping model, multiple clamping points are defined based on the object to be clamped, and fixed-point clamping of multiple robotic arms is triggered according to the multiple clamping points, and dynamic balance is performed based on the clamping forces applied by the multiple robotic arms, including: In the clamping model, the spatial position of the object to be clamped is collected; Perform multi-directional shooting based on the spatial position of the object to be clamped, and output multiple images of the object to be clamped in different directions; A three-dimensional model of the object to be clamped is constructed based on the multiple images, and multiple clamping points are defined according to the three-dimensional model of the object to be clamped. At this time, the three-dimensional model of the object to be clamped performs a dynamic drop simulation in the virtual scene and outputs multiple clamping points; Triggering fixed-point clamping of multiple robotic arms according to multiple clamping points, at this time, matching the corresponding robotic arms based on the position and orientation of the clamping points; Collect the clamping force applied by each clamping point, and construct a force distribution map of the object to be clamped based on multiple clamping forces; When the object to be clamped moves with the movement of the industrial robot, the change parameters in the force distribution diagram are monitored, and the dynamic adjustment of the corresponding robot arm is triggered based on the change amount of the change parameters to ensure the dynamic balance of the object to be clamped during the movement.

2. The collaborative method of industrial robots under multi-sensory fusion according to claim 1, characterized in that: The collecting of multiple mechanical arms in the industrial robot and associating the multiple mechanical arms include: Locate industrial robots and respond to them in all directions; Obtain response results, and determine the positions and operating ranges of multiple robotic arms in the response results; Collect the positions of multiple robotic arms; Defining relative positions of the plurality of robotic arms based on the positions of the plurality of robotic arms, and constructing an operation space according to the relative positions and the operation range; Multiple robotic arms are associated, and the degree of coordination of the multiple robotic arms in the associated state is trained. At this time, the operation points are arranged in sequence along the circular direction in the operation space, and the coordination relationship of the multiple robotic arms is triggered based on the operation points.

3. The collaborative method of industrial robots under multi-sensory fusion according to claim 2 is characterized in that: Each robot arm is equipped with an environmental sensing component, and the environmental parameters in different directions are sensed by the environmental sensing component, including: Each robot arm is equipped with an environment sensing component. At this time, the environment sensing component is located on one side of the operating end of the robot arm and is arranged at a distance from the operating end; Collect the current direction of the operation terminal; Adjust the orientation of the sensing end of the environment sensing component based on the current orientation of the operating end, and collect the environment images output by each environment sensing component on the same robotic arm; forming a perception image based on the synthesis of the environmental images, and marking the position of the operating end in the perception image; If the position of the operating end is not at the center of the perceived image, the offset between the position of the operating end and the center is collected, and each environmental sensing element is adjusted according to the offset until the position of the operating end is at the center of the perceived image; The environment sensing element senses environmental parameters in different directions and marks the environmental parameters on the outer peripheral side of the operating end.

4. The collaborative method of industrial robots under multi-sensory fusion according to claim 3 is characterized in that: In the process of executing the collaborative instruction, the multiple robotic arms are dynamically regulated during the working process and adjusted according to the environmental parameters collected in real time, including: In the process of executing the collaborative instruction, multiple robotic arms move along the corresponding sub-action instructions; Monitor the movements of multiple robotic arms in real time and collect the corresponding movement routes.

5. The collaborative method of industrial robots under multi-sensory fusion according to claim 4 is characterized in that: In the process of executing the collaborative instruction, the multiple robotic arms are dynamically regulated during the working process and adjusted according to the environmental parameters collected in real time, and also include: Perform environmental perception along the action route and define the surrounding environment of the action route; The activity of the robot arm in the action route is defined according to the surrounding environment of the action route, and the action of the robot arm is adjusted according to the environmental parameters collected in real time.

6. The collaborative method of industrial robots under multi-sensory fusion according to claim 5, characterized in that: The step of determining keywords based on collaborative instructions and constructing a gripping model based on the keywords and multiple robotic arms includes: Collect collaborative instructions; Determine keywords based on collaborative instructions and define the action types of keywords; Associate the corresponding robot arm according to the action type of the keyword; If the action type of the keyword is a gripping type, a gripping model is constructed based on multiple robotic arms; In the clamping model, construct the active and passive relationships of multiple robotic arms; The coordination of multiple robotic arms in the clamping process is triggered according to the active relationship and the slave relationship, and the coordination efficiency between the multiple robotic arms is detected in real time. At the same time, the coordination efficiency between the multiple robotic arms is regulated according to the response logic between the multiple robotic arms.

7. A collaborative system of industrial robots under multi-sensory fusion, characterized in that: The collaborative system under multi-sensory fusion of the industrial robot is applied to the collaborative method under multi-sensory fusion of the industrial robot as claimed in any one of claims 1 to 6, and the collaborative system under multi-sensory fusion of the industrial robot includes: A collection module, used to collect data from multiple robotic arms in an industrial robot and associate the multiple robotic arms; Environmental parameter module, used to equip each robot arm with an environmental sensing component, and sense environmental parameters in different directions through the environmental sensing component; An environment result module, used to form an environment parameter set from multiple environment parameters, and match environment results based on the environment parameter set; A coordination instruction module is used to output coordination instructions of multiple robot arms according to environmental results and acquired motion instructions; The dynamic control module is used to dynamically control multiple robotic arms during the execution of collaborative instructions and make adjustments based on environmental parameters collected in real time; A gripping module, used to determine keywords based on collaborative instructions and build a gripping model based on the keywords and multiple robotic arms; The dynamic balancing module is used to define multiple clamping points based on the object to be clamped in the clamping model, trigger the fixed-point clamping of multiple robotic arms according to the multiple clamping points, and perform dynamic balancing based on the clamping forces applied by the multiple robotic arms.

Images