Mechanical arm segmentation path autonomous evolution method based on livestock carcass segmentation big data and double-arm robot livestock carcass segmentation system

Abstract

A kind of mechanical arm segmentation path autonomous evolution method based on livestock carcass segmentation big data and double-arm robot livestock carcass segmentation system, initial training stage video edge processor according to the walking speed of 3D video information fusion conveyor belt transmitted by 1#, 2#RGBD camera to meat piece three-dimensional structure modeling, obtain the 3D model with color texture characteristics;And according to the 2D operation process video information transmitted by 1#, 2# industrial camera, the segmentation process of operator is recorded;Algorithm server reads video server data, learns according to 3D model and operator operation process video information, extracts corresponding segmentation path, obtains "cutting trajectory extraction model";Then the path information given by 3D model and "cutting trajectory extraction model" is used as training sample set, and "cutting trajectory planning generation model" is obtained by training;Using evolution stage double-arm robot, according to the cutting path planned by "cutting trajectory planning generation model", simulate the operation of operator to complete the segmentation of meat piece.

Description

TECHNICAL FIELD

[0001] The present application relates to a meat finishing, cutting process robot end operation trajectory planning method, in particular to a dual-arm robot cutting trajectory autonomous co-evolution method based on meat cutting big data and a dual-arm robot livestock carcass cutting system. BACKGROUND

[0002] At present, in the world, in the process of livestock slaughtering, the finishing and cutting process of meat still cannot do without a large number of manual participation. The whole finishing and cutting work is faced with the problems of high intensity, poor environment, etc., and the meat finishing post is faced with the predicament of difficult recruitment, and even will face the situation of no one available. On the other hand, with the improvement of social living standards, the people have put forward higher requirements for pork supply. Many countries have also introduced more stringent laws on meat production safety, hygiene conditions and working environment, which makes the traditional slaughtering industry face great challenges. Therefore, it is particularly urgent to develop a robot system for the cutting process of the slaughtering industry based on artificial intelligence technology. At present, the control algorithm of the robot has been very mature, and robots of various brands can realize the accurate operation of any trajectory in the working space with very high precision.

[0003] However, due to the existence of factors such as size, placement position, meat quality composition of meat products on the production line, the planning of the robot end operation trajectory is a very difficult problem to solve. Table 1 gives the statistics of the existing related technologies.

[0004] Table 1. List of livestock carcass cutting patents in recent years

[0005]

[0006] From the statistics in the table, it can be seen that the research on livestock carcass cutting is a relatively popular research field. Especially in the past three years, there are many technical methods proposed. Looking at various methods, most of them are concentrated on image or point cloud data recognition and segmentation. However, there are few reports on modeling using existing cutting big data.

[0007] In order to overcome the influence of the above factors, solve the problem of robot end operation trajectory planning, and overcome the precision interference brought by the above influencing factors, which is directly related to the popularization and application of automatic technology for robot completely replacing manual work. SUMMARY

[0008] The present application proposes a mechanical arm cutting path autonomous evolution method based on livestock carcass cutting big data to overcome the shortcomings of the prior art. Based on big data technology, point cloud data modeling technology, AI generated model and reinforcement learning algorithm, the robot end operation trajectory of the finishing and cutting process is planned, and the planning precision of the robot end operation trajectory is improved.

[0009] The technical scheme adopted by the present application is as follows:

[0010] A mechanical arm segmentation path autonomous evolution method based on livestock carcass segmentation big data, comprising an initial training stage and a use evolution stage; the process of the initial training stage comprises:

[0011] Step S1.1, a first and a second RGBD camera and a first and a second industrial camera are installed around the segmentation operator in the conveying belt direction to obtain the on-site video signal input into the video edge processor deployed on site;

[0012] Step S1.2, the video edge processor fuses the walking speed of the conveying belt according to the 3D video information transmitted by the first and second RGBD cameras, and performs three-dimensional structure modeling on the meat blocks conveyed on the conveying belt to obtain a 3D model with color texture characteristics, and uploads the 3D model to a video server for storage;

[0013] Step S1.3, the video edge processor records the segmentation process of the operator (including the left and right hand poses of the operator, the running path of the cutter, the relative position relationship between the cutter and the meat block, and the three-dimensional model of the cut object) according to the 2D video information transmitted by the first and second industrial cameras and the 3D model with color texture characteristics established in step S1.2, and uploads the recorded operation process of the operator to the video server for storage;

[0014] Step S1.4, an algorithm server is provided, which reads the previously stored data from the video server and completes the training of the related algorithm model, which comprises:

[0015] Step S1.4-1, according to the 3D model with color texture characteristics of the meat block and the operation process video information of the operator, the corresponding segmentation path is learned and extracted to obtain a "cutting trajectory extraction model";

[0016] Step S1.4-2, the path information given by the 3D model and the "cutting trajectory extraction model" is taken as a training sample set, and a "cutting trajectory planning generation model" is trained;

[0017] The above-mentioned "cutting trajectory extraction model" and "cutting trajectory planning generation model" are stored in the algorithm server;

[0018] In the use evolution stage, the "cutting trajectory planning generation model" stored in the algorithm server is deployed in the video edge processor to plan the cutting path of the dual-arm robot in the operation area to replace manual operation, simulate the operation mode of the operator, and complete the segmentation of the meat block.

[0019] The mechanical arm cutting path autonomous evolution method based on large data of livestock carcass segmentation, uses an evolution stage, and the process includes:

[0020] Firstly, the "cutting trajectory planning generation model" completes the planning of the left and right hand cutting paths of the dual-arm robot (including the running posture of the left and right hands of the robot, the running trajectory of the cutter, etc.) according to the constructed meat block 3D model with color texture characteristics;

[0021] Then, the video edge processor sends the planned cutting path to the dual-arm robot through the interface, and the dual-arm robot cuts the meat block according to the planned running trajectory;

[0022] In the process of cutting according to the planned running trajectory, the dual-arm robot performs closed-loop control on the cutting route according to the end cutting force feedback, and the end cutting force feedback is obtained by the method of estimating the current of each joint.

[0023] The mechanical arm cutting path autonomous evolution method based on large data of livestock carcass segmentation, in step S1.4-1, the modeling process of the cutting trajectory generation model includes the following five steps:

[0024] (1) Data preprocessing:

[0025] Firstly, each video is segmented by frame, and continuous action frame sequences are extracted, and each frame is labeled using a labeling tool; the network model is trained using the labeled data to automatically detect and track the position of the cutter in each frame;

[0026] Then, the multiple key points of the cutter are detected, the angle of the cutter relative to the horizontal line is calculated, and the calculated angle value of each frame is stored together with the position information;

[0027] Next, visualize the cutter trajectory and angle, check the continuity of the trajectory, identify and correct abnormal points, confirm the reasonableness of the cutting angle, and avoid extreme or impossible angle values;

[0028] Finally, the three-dimensional pork model is meshed, and the mesh is exported in the required format for subsequent physical simulation and spatial layout analysis;

[0029] (2) Feature extraction and action recognition:

[0030] Firstly, the convolutional neural network is used to extract image features of each frame in the video, and the motion features and cutting angle information of the cutter are obtained;

[0031] Subsequently, the time sequence model is used to model the continuous action frames to capture the time sequence relationship of the actions; then the output of the time sequence model is classified by the action classification algorithm classifier to identify different cutting action patterns;

[0032] (3) Simulation of three-dimensional cutting process:

[0033] The interaction between the cutting tool and the 3D pork model is simulated using a physics engine to ensure that the cutting path conforms to physical constraints; the position and angle changes of the cutting tool at different cutting stages are understood through 3D spatial analysis; a path planning algorithm is used to generate the cutting path in 3D space and collision detection is performed to ensure the feasibility and safety of the planned path.

[0034] (4) Model training and mapping relationship learning:

[0035] Supervised learning methods are used to learn the mapping relationship between cutting actions and cutting positions based on labeled cutting action and position data; through transfer learning, the cutting patterns learned from the video are transferred to three-dimensional space to ensure that the model can make accurate cutting predictions on the three-dimensional pork model;

[0036] (5) Cutting trajectory extraction:

[0037] Given a new 3D pork model, a preset starting position, and a cutting target, the model predicts and extracts an optimal cutting path based on the input parameters.

[0038] The autonomous evolution method for robotic arm segmentation paths based on large-scale animal carcass segmentation data, in step S1.4-2, the modeling process of the cutting trajectory planning model includes an initial training phase and an evolutionary phase:

[0039] Initial training phase:

[0040] First, the 3D point cloud data containing color information of the 3D model is downsampled to reduce the amount of computation. Then, features are extracted from it, and a certain number of feature points are retained as a set of standard samples. The sample contains three-dimensional position coordinates and RGB channel color information, i.e. n 6-dimensional data. The label path data adopts the straight line fitting method of Huber regression. Considering that the cutting point, blade posture and movement distance need to be determined when cutting, the data is finally converted into a 7-dimensional format.

[0041] In the initial training phase, the data is passed through a fully connected input layer, and then sequentially through several multi-layer network blocks composed of convolution, pooling, activation, and normalization layers. Finally, the output layer outputs the calculation results. The predicted path data and label path data of the output layer are used to calculate the loss for gradient update and iteratively train the model.

[0042] Using the evolutionary stage: Adjusting the regularization term of the model's loss function and adding a feedback scoring component from the evolutionary module makes the path trajectory generated by the model more reasonable and effective in actual robot operation.

[0043] The mechanical arm cutting path autonomous evolution method based on livestock carcass cutting big data adopts an "evolution module based on reinforcement learning" to optimize the "cutting trajectory planning generation model", the video information of the operation process of the robot is forwarded to the video server by the video edge processor for optimization learning by the model in the algorithm server, and the "cutting trajectory planning generation model" is evolved according to the operation video of the robot and the corresponding scoring condition; at the appropriate time, the model optimized by the algorithm server is deployed again in the video edge processor, and the iteration of the whole control system is completed.

[0044] The mechanical arm cutting path autonomous evolution method based on livestock carcass cutting big data uses a reinforcement learning algorithm to optimize the meat cutting trajectory:

[0045] Firstly, the robot needs to extract the shape, size, cutting area and other state features of the meat block according to the environmental information of the 3D point cloud data and image data, and the pre-processing stage converts these data into state information available for learning;

[0046] The reinforcement learning algorithm defines the state that the robot can perceive and the action that it can perform, defines the state space and the action space, the state space includes the position of the robot cutter, the state of the cutting tool, the hardness and the cutting part; the action space includes adjusting the angle of the cutting tool, changing the cutting path, adjusting the cutting depth and speed, etc., the robot selects the best action from the current state through continuous exploration and utilization;

[0047] The reinforcement learning algorithm changes the selection of robot actions by the different force values of soft meat and bone feedback, thereby changing the trajectory path, and uses the position error of the robot cutter and the workbench to determine whether the meat block is cut, the network result prediction of the force feedback value and the position error value of each action of the robot state, then gives positive and negative rewards to each action, selects the action with the maximum reward, that is, the optimal cutting path, thereby guiding the robot to operate more accurately and efficiently;

[0048] During the training process, the planned trajectory learns through continuous interaction with the cutting process, and the strategy network is updated continuously, so that the robot can optimize the cutting path through continuous attempts, and gradually approaches the optimal solution through multiple iterations.

[0049] The mechanical arm segmentation path autonomous evolution method based on livestock carcass segmentation big data adopts a dual-arm robot cooperative evolution method, for a plurality of dual-arm robots continuously arranged on a pipeline, wherein each dual-arm robot is equipped with a video information processor, which processes video signals of four cameras arranged in the area where the dual-arm robot is located, and drives two mechanical arms of the dual-arm robot to move; the meat block moves from the first processor to the rear on the pipeline, and when the first dual-arm robot finishes processing the meat block, the processed information is transmitted to the second processor as a reference for path planning, and the above information is transmitted to the rear one by one, realizing cooperative evolution of path planning of subsequent mechanical arms, so as to improve the accuracy of the meat block cutting path.

[0050] A dual-arm robot livestock carcass segmentation system based on livestock carcass segmentation big data, comprising:

[0051] Data acquisition device:

[0052] A first RGBD camera and a second RGBD camera are arranged in front of and behind the left hand of the operator to record video information of the meat block passing through the conveying belt; a first industrial camera and a second industrial camera are arranged in front of and behind the right hand of the operator to record video information of the segmentation process of the operator; and the video signals of the four cameras are input into a video edge processor arranged on site.

[0053] Video edge processor and video server:

[0054] According to the 3D video information transmitted by the first RGBD camera and the second RGBD camera, the walking speed of the conveying belt is fused, and a three-dimensional structure model of the meat block conveyed on the conveying belt is formed, thereby forming a three-dimensional structure model with color texture characteristics;

[0055] According to the 2D video information transmitted by the first industrial camera and the second industrial camera, the segmentation process of the operator is recorded;

[0056] The three-dimensional structure model of the meat block with color texture and the operation process of the operator are uploaded to the video server;

[0057] The video edge processor is composed of an ARM processor and an FPGA chip; the ARM processor runs a Linux operating system to complete scheduling of various functions; and the FPGA chip realizes running of an artificial intelligence algorithm;

[0058] Algorithm server:

[0059] The stored data in the video server is read to complete training of an algorithm model related to a "cutting trajectory generation model" and a "cutting trajectory planning model";

[0060] Dual-arm robot:

[0061] The algorithm server deploys the stored "cutting trajectory planning model" to the video edge processor; the video edge processor sends to the dual-arm robot through the interface, and the dual-arm robot simulates the operation mode of the operator to complete the cutting of the meat block according to the planned trajectory.

[0062] The dual-arm robot livestock carcass cutting system includes two mechanical hands, one large and one small, wherein the large mechanical hand completes the left hand function, mainly including some auxiliary actions and heavy load work; the small mechanical hand completes the right hand function, mainly including relatively complex cutting actions.

[0063] The dual-arm robot livestock carcass cutting system, the video edge processor communicates data with the video server and the algorithm server through wifi or industrial ethernet; in the cutting and bone removal process, closed-loop control is performed on the route according to the end cutting force feedback, wherein the end force feedback is realized by current estimation of each joint of the mechanical hand and fusion of video data calculation.

[0064] The application has the following beneficial effects:

[0065] 1. The mechanical arm cutting path autonomous evolution method based on livestock carcass cutting big data, based on big data technology, point cloud data modeling technology, artificial intelligence technology, etc., proposes a universal fine cutting process robot end running trajectory planning and cooperative evolution method, which avoids the interference of factors such as size, placement position, and meat composition of meat products on the production line on the robot end running trajectory planning, and improves the planning accuracy of the robot end running trajectory.

[0066] 2. The mechanical arm cutting path autonomous evolution method based on livestock carcass cutting big data, the video information of the operator's operation process is forwarded to the video server by the video edge processor for optimization learning by the model in the algorithm server. The optimization process adopts an "evolution module based on reinforcement learning", and the "cutting trajectory planning generation model" is evolved according to the operation video of the robot and the corresponding scoring. At the right time, the model optimized by the algorithm server is deployed to the video edge processor again, completing the iteration of the entire control system.

[0067] 3. The mechanical arm cutting path autonomous evolution method based on livestock carcass cutting big data adopts a dual-arm robot with left and right hands, completely simulates the operation mode of the operator, can adapt to cutting procedures of any part, and at the same time reduces the development cost of multiple hardware.

[0068] 4. The application is based on the big data of livestock carcass segmentation, the control program of the mechanical arm is implanted in the video edge processor, and the trajectory given by the "cutting trajectory planning generation model" is used for cutting. In the process of cutting, in the process of bone picking, the closed loop control is realized by relying on the tactile feedback of the end. The tactile feedback of the end is obtained by detecting the current of each joint of the manipulator. Compared with the scheme using tactile sensor, it has the advantages of low cost and wide perception range.

[0069] 5. The application is based on the big data of livestock carcass segmentation, the information between the processes is transmitted backward, the one-way sharing of cutting data and meat structure is realized, and the cutting accuracy of the whole pipeline is improved. BRIEF DESCRIPTION OF DRAWINGS

[0070] Figure 1 The technical roadmap of the double-arm robot meat block segmentation path autonomous planning method of the application is shown;

[0071] Figure 2 The data acquisition scheme of the double-arm robot meat block segmentation path autonomous planning method of the application is shown;

[0072] Figure 3 The cutting trajectory generation model of the double-arm robot meat block segmentation path autonomous planning method of the application is shown;

[0073] Figure 4 The cutting trajectory planning model of the double-arm robot meat block segmentation path autonomous planning method of the application is shown;

[0074] Figure 5 The double-arm robot control diagram of the double-arm robot meat block segmentation path autonomous planning method of the application is shown;

[0075] Figure 6 The collaborative evolution method of the double-arm robot meat block segmentation path autonomous planning method of the application is shown. DETAILED DESCRIPTION

[0076] In order to make the technical concept and advantages of the application realize its invention purpose more clear and clear, the technical scheme of the application is further described in detail below in combination with the drawings. It should be understood that the following examples are only used to explain and illustrate the preferred embodiments of the application, and should not be regarded as and constitute a limitation on the scope of the patent protection claimed by the application. EMBODIMENT

[0077] Referring to Figure 1 The application is based on the big data of livestock carcass segmentation, and includes an initial training stage and a use evolution stage; the initial training stage includes:

[0078] Step S1.1, install the first and second RGBD cameras and the first and second industrial cameras around the cutting operation worker in the conveying belt direction, and acquire the on-site video signal to input the video edge processor deployed on site;

[0079] Step S1.2, the video edge processor fuses the walking speed of the conveying belt according to the 3D video information transmitted by the first and second RGBD cameras, and performs three-dimensional structure modeling on the meat blocks conveyed on the conveying belt to obtain a 3D model with color texture characteristics, and uploads the 3D model to the video server and stores it;

[0080] Step S1.3, the video edge processor records the operation process of the operation worker (including the left and right hand poses of the operation worker, the running path of the cutter, the relative position relationship between the cutter and the meat block, and the three-dimensional model of the cut object) according to the 2D video information transmitted by the first and second industrial cameras and the 3D model with color texture characteristics established in step S1.2, and uploads the recorded operation process of the operation worker to the video server and stores it;

[0081] Step S1.4, an algorithm server is provided, which reads the previously stored data from the video server and completes the training of the related algorithm model, which includes:

[0082] Step S1.4-1, according to the 3D model with color texture characteristics of the meat block and the operation process video information of the operation worker, learning and extracting the corresponding segmentation path to obtain a "cutting trajectory extraction model";

[0083] Step S1.4-2, taking the path information given by the 3D model and the "cutting trajectory extraction model" as a training sample set, and training to obtain a "cutting trajectory planning generation model";

[0084] The above-mentioned "cutting trajectory extraction model" and "cutting trajectory planning generation model" are stored in the algorithm server;

[0085] In the use evolution stage, the "cutting trajectory planning generation model" stored in the algorithm server is deployed in the video edge processor to plan the cutting path of the artificial double-arm robot in the operation area, and simulate the operation of the operation worker to complete the segmentation of the meat block. Embodiment

[0086] The embodiment is based on the mechanical arm segmentation path autonomous evolution method of livestock carcass segmentation big data, which is different from embodiment 1. Further, the process of the use evolution stage includes:

[0087] Firstly, the "cutting trajectory planning generation model" completes the planning of the left and right hand cutting paths of the dual-arm robot (including the running posture of the left and right hands of the robot, the running trajectory of the cutter, etc.) according to the 3D model with color texture features of the constructed meat block;

[0088] Then, the video edge processor sends the planned cutting path to the dual-arm robot through the interface, and the dual-arm robot cuts the meat block according to the planned cutting path;

[0089] During the cutting process according to the planned cutting path, the dual-arm robot performs closed-loop control on the cutting route according to the end cutting force feedback, and the end cutting force feedback is obtained by using the joint current estimation method. Embodiment

[0090] The embodiment is based on the mechanical arm cutting path autonomous evolution method of livestock carcass cutting big data, which is different from embodiment 1 and embodiment 2: in step S1.4-1, the modeling process of the cutting trajectory generation model includes the following five steps:

[0091] (1) Data preprocessing:

[0092] Firstly, each video is segmented by frame, and continuous action frame sequences are extracted, and each frame is labeled using a labeling tool; the labeled data is used to train a network model to automatically detect and track the position of the cutter in each frame;

[0093] Then, the positions of multiple key points of the cutter are detected, the angle of the cutter relative to the horizontal line is calculated, and the calculated angle value of each frame is stored together with the position information;

[0094] Next, visualize the cutter trajectory and angle, check the continuity of the trajectory, identify and correct abnormal points, confirm the reasonableness of the cutting angle, and avoid extreme or impossible angle values;

[0095] Finally, the three-dimensional pork model is meshed, and the mesh is exported in the required format for subsequent physical simulation and spatial layout analysis;

[0096] (2) Feature extraction and action recognition:

[0097] Firstly, a convolutional neural network is used to extract image features from each frame in the video, and the motion features and cutting angle information of the cutter are obtained;

[0098] Subsequently, a time series model is used to model the continuous action frames to capture the temporal relationship of the actions; then, an action classification algorithm classifier is used to classify the output of the time series model to identify different cutting action patterns;

[0099] (3) Three-dimensional cutting process simulation:

[0100] The interaction of the knife and the three-dimensional pork model is simulated using a physics engine to ensure that the cutting path conforms to physical constraints; the position and angle changes of the knife at different cutting stages are understood through three-dimensional space analysis; a path planning algorithm is used to generate a cutting path in three-dimensional space and perform collision detection to ensure the feasibility and safety of the planned path;

[0101] (4) Model training and mapping relationship learning:

[0102] A supervised learning method is used to learn the mapping relationship between cutting actions and cutting positions based on labeled cutting action and position data; transfer learning is used to transfer the cutting patterns learned from videos to three-dimensional space to ensure that the model can make accurate cutting predictions on three-dimensional pork models;

[0103] (5) Cutting trajectory extraction:

[0104] Given a new three-dimensional pork model, a new starting position and a cutting target, the model predicts and extracts an optimal cutting path based on the input parameters. Embodiment

[0105] The mechanical arm cutting path autonomous evolution method based on large data of livestock carcass segmentation described in this embodiment is different from the aforementioned embodiments in that in step S1.4-2, the modeling process of the cutting trajectory planning model includes:

[0106] (1) Initial training stage:

[0107] First, the 3D point cloud data containing color information of the 3D model is down-sampled to reduce the amount of calculation, then the features are extracted, and a certain number of feature points are retained as a set of standard samples; the sample contains three-dimensional position coordinates and RGB channel color information, i.e. n 6-dimensional data; the label path data uses Huber regression linear fitting method, considering the determination of the knife point, blade pose and movement distance when cutting down, finally converting the data into 7-dimensional format;

[0108] In the initial training stage, after the data passes through the fully connected input layer, it successively passes through several multi-layer network blocks composed of convolution, pooling, activation and normalization layers, and finally outputs the calculation result from the output layer; the prediction path data of the output layer is calculated with the label path data for gradient update, and the model is iteratively trained;

[0109] (2) Use evolution stage: adjust the regularization term of the model Loss function, add the feedback score part from the evolution module, so that the path trajectory generated by the model is more reasonable and effective in actual robot operation. Embodiment

[0110] The mechanical arm cutting path autonomous evolution method based on large data of livestock carcass cutting described in this embodiment is different from the previous embodiments in that: the "evolution module based on reinforcement learning" is used to optimize the "cutting trajectory planning model", the video information of the operation process of the robot is forwarded to the video server by the video edge processor for optimization learning by the model in the algorithm server, and the "cutting trajectory planning model" is evolved according to the operation video of the robot and the corresponding scoring; at the appropriate time, the model optimized by the algorithm server is deployed again to the video edge processor to complete the iteration of the entire control system.

[0111] In the modeling process of the cutting trajectory generation model, the meat block cutting trajectory is optimized using a reinforcement learning algorithm, and the process is as follows:

[0112] First, the robot needs to extract the shape, size, and segmentation area of the meat block according to the environmental information of the 3D point cloud data and image data, and the pre-processing stage converts these data into state information that can be learned;

[0113] The reinforcement learning algorithm clearly defines the states that the robot can perceive and the actions that it can perform, defines the state space and action space, and the state space includes the position of the robot's cutter, the state of the cutting tool, the hardness, and the cutting site. The action space includes adjusting the angle of the cutting tool, changing the cutting path, adjusting the cutting depth and speed, etc. The robot selects the best action from the current state through continuous exploration and utilization;

[0114] The reinforcement learning algorithm changes the selection of robot actions by the different force values of soft meat and bone feedback, thereby changing the trajectory path, and uses the position error of the robot cutter and the workbench to determine whether the meat block is cut, and the network result prediction of the force feedback value and the position error value of each action of the robot state. Then, positive and negative rewards are given to each action, and the action with the maximum reward, that is, the optimal cutting path, is selected to guide the robot to operate more accurately and efficiently;

[0115] During the training process, the planned trajectory learns through continuous interaction with the cutting process, and the policy network is updated continuously, so that the robot can optimize the cutting path through continuous attempts, and gradually approaches the optimal solution through multiple iterations. EMBODIMENT

[0116] The embodiment is based on the big data of livestock carcass segmentation, and the dual-arm robot cooperative evolution method is used. For the multiple groups of dual-arm robots arranged continuously on the pipeline, each dual-arm robot is equipped with a video information processor, which processes the video signals of the four cameras arranged in the area where the dual-arm robot is located, and drives the two mechanical arms of the dual-arm robot to move; the meat block moves from the first processor to the rear on the pipeline, and when the first dual-arm robot finishes processing the meat block, the processed information is transmitted to the second processor as a reference for path planning, and the above information is transmitted to the rear one by one, realizing the cooperative evolution of the path planning of the subsequent mechanical arms, so as to improve the accuracy of the meat cutting path. Embodiment

[0117] Reference Figures 1-6 The application provides a dual-arm robot livestock carcass segmentation system based on the big data of livestock carcass segmentation, which comprises:

[0118] Data acquisition device:

[0119] The first RGBD camera and the second RGBD camera are arranged in front of and behind the left hand of the operator, and are used for recording the video information of the meat block passing through the conveying belt; the first industrial camera and the second industrial camera are arranged in the upper front position and the lower rear position of the right hand of the operator, and are used for recording the segmentation process video information of the operator; the video signals of the four cameras are input into the video edge processor arranged on site;

[0120] Video edge processor and video server:

[0121] According to the 3D video information transmitted by the first RGBD camera and the second RGBD camera, the walking speed of the conveying belt is fused, and the three-dimensional structure model of the meat block conveyed on the conveying belt is formed, so that the three-dimensional structure model with color texture characteristics is formed;

[0122] According to the 2D video information transmitted by the first industrial camera and the second industrial camera, the segmentation process of the operator is recorded;

[0123] The above-mentioned three-dimensional structure model of the meat block with color texture and the operation process of the operator are uploaded to the video server;

[0124] The video edge processor is composed of an ARM processor and an FPGA chip; the ARM processor runs a Linux operating system and completes the scheduling of various functions; and the FPGA chip realizes the running of an artificial intelligence algorithm;

[0125] Algorithm server:

[0126] Read the stored data from the video server, complete the training of the "cutting trajectory extraction model" and the "cutting trajectory planning generation model" algorithm model;

[0127] Double-arm robot:

[0128] The algorithm server deploys the stored "cutting trajectory planning generation model" to the video edge processor; the video edge processor sends to the double-arm robot through the interface, and the double-arm robot simulates the operation mode of the operator to complete the segmentation of the meat block according to the planned trajectory.

[0129] The double-arm robot includes two mechanical hands, one large and one small, wherein the large mechanical hand completes the left hand function, mainly including some auxiliary actions and heavy load work; the small mechanical hand completes the right hand function, mainly including relatively complex cutting actions.

[0130] The video edge processor communicates data with the video server and the algorithm server through wifi or industrial ethernet; in the process of cutting and bone removal, the route is closed loop controlled according to the end cutting force feedback, wherein the end force feedback is realized by estimating the current of each joint of the mechanical hand and fusing the video data to calculate. Embodiment

[0131] The present application is based on the mechanical arm segmentation path autonomous evolution method of livestock carcass segmentation big data and the double-arm robot livestock carcass segmentation system based on the method, and the technical route is as shown in Figure 1 .

[0132] The whole method is divided into initial training stage and use evolution stage.

[0133] In the initial training stage, two RGBD cameras and two industrial cameras are installed around the segmentation operator. The video signals of the four cameras are input to the video edge processor deployed on site. The video edge processor is composed of an ARM processor and an FPGA chip. The Linux operating system runs in the ARM processor to complete the scheduling of various functions. The FPGA chip realizes the running of artificial intelligence algorithms. According to the 3D video information transmitted by the first RGBD camera and the second RGBD camera, the video edge processor fuses the walking speed of the conveyor belt, and performs three-dimensional structure modeling on the meat block conveyed on the conveyor belt, forming a three-dimensional structure model with color texture characteristics. Then, according to the 2D video information transmitted by the first industrial camera and the second industrial camera, and the three-dimensional model of the meat block just established, the video edge processor records the segmentation process of the operator (including the left and right hand poses of the operator, the running path of the knife, the relative position relationship between the knife and the meat block, and the three-dimensional model of the cut object, etc.). The above-mentioned three-dimensional structure model of the meat block with color texture and the operation process of the operator are uploaded to the video server for use.

[0134] The algorithm server is equipped to read the previously stored data from the video server and complete the training of the relevant algorithm model. It includes: (1) learning to extract the corresponding segmentation path according to the colored texture 3D model of the meat block and the operator's operation process video information, to obtain the "cutting trajectory extraction model"; (2) taking the 3D model and the path information given by the "cutting trajectory extraction model" as the training sample set, to train and obtain the "cutting trajectory planning model".

[0135] The above-mentioned "cutting trajectory extraction model" and "cutting trajectory planning generation model" are stored in the algorithm server.

[0136] In the evolution stage, the "cutting trajectory planning generation model" on the algorithm server is deployed in the video edge processor. At this time, a dual-arm robot is installed in the operation area to replace manual work and complete the segmentation of the meat block. Here, a dual-arm robot with left and right hands is used to completely simulate the operation mode of the operator, which can adapt to the segmentation process of any part and reduce the development cost of multiple hardware. First, the "cutting trajectory planning generation model" completes the planning of the right-hand cutting path of the dual-arm robot according to the constructed meat block with texture three-dimensional model (including the running posture of the robot left and right hands, the running trajectory of the cutter, etc.). Then, the planned path is sent to the dual-arm robot through the interface to complete the segmentation of the meat block. During the cutting process according to the planned trajectory, the dual-arm robot needs to perform closed-loop control on the route according to the end cutting force feedback. The end force feedback is obtained by using the method of estimating the current of each joint.

[0137] The video information of the above process is forwarded by the video edge processor to the video server for optimization learning by the model in the algorithm server. The optimization process uses an "evolution module based on reinforcement learning" to evolve the "cutting trajectory planning model" according to the operation video of the robot and the corresponding scoring. At the appropriate time, the optimized model of the algorithm server is deployed again to the video edge processor to complete the iteration of the entire control system.

[0138] Data acquisition scheme and data acquisition device

[0139] Figure 2 The data acquisition scheme in the initial training stage is shown. The operator stands on one side of the conveyor belt during work. The conveyor belt moves from the left hand side to the right hand side of the operator. Equipment supports are arranged on both sides of the operator for installing data acquisition equipment. The first and second RGBD cameras are installed in front of and behind the left hand side of the operator. The two cameras shoot obliquely downward to record the meat blocks passing along the conveyor belt. The first and second industrial cameras are installed in front of and behind and above and below the right hand side of the operator. The two cameras shoot obliquely downward to record the segmentation process of the operator. A video edge processor is installed above the equipment support.

[0140] In actual operation, the conveyor belt can move from the right side of the operator to the left side. At this time, the first and second RGBD cameras are installed on the bracket on the right side of the operator, and the first and second industrial cameras are installed on the bracket on the left side of the operator. The video edge processor communicates data with the video server and the algorithm server through wifi or industrial Ethernet line. The power line and network cable on site are connected with the equipment bracket through hoisting or ground wiring.

[0141] Cutting trajectory extraction model

[0142] Figure 3 The cutting trajectory generation model of the dual-arm robot meat segmentation path autonomous planning method is shown. The generation of the cutting trajectory is divided into five steps: data preprocessing, feature extraction and action recognition, three-dimensional cutting process simulation, model training and mapping relationship learning, and cutting trajectory generation, wherein:

[0143] (1) Data preprocessing. First, each video is segmented by frame, and continuous action frame sequences are extracted. Each frame is labeled using a labeling tool. The network model is trained using the labeled data to automatically detect and track the position of the knife in each frame. Then, using multiple key points of the detected knife, such as the knife tip and handle, the angle of the knife relative to the horizontal line is calculated, and the calculated angle value of each frame is stored together with the position information. Next, visualize the knife trajectory and angle, check the continuity of the trajectory, identify and correct abnormal points, confirm the reasonableness of the cutting angle, and avoid extreme or impossible angle values. These labeled data will provide high-quality input for subsequent deep learning model training. Finally, the three-dimensional pork model is meshed, and the mesh is exported in the required format for subsequent physical simulation and spatial layout analysis.

[0144] (2) Feature extraction and action recognition. Use a convolutional neural network to extract image features from each frame in the video, obtaining the motion characteristics of the knife and the cutting angle information. Then, use a time series model to model the continuous action frames to capture the temporal relationship of the actions. Classify the output of the time series model using an action classification algorithm to identify different cutting action patterns.

[0145] (3) Three-dimensional cutting process simulation. Use a physics engine to simulate the interaction between the knife and the three-dimensional pork model to ensure that the cutting path meets the physical constraints. Understand the position and angle changes of the knife at different cutting stages through three-dimensional space analysis. Use path planning algorithms to generate cutting paths in three-dimensional space and perform collision detection to ensure the feasibility and safety of the planned path.

[0146] (4) Model training and mapping relationship learning. Using supervised learning method based on the labeled cutting action and position data, learning the mapping relationship between cutting action and cutting position. Through transfer learning, the cutting mode learned from the video is transferred to the three-dimensional space, ensuring that the model can make accurate cutting prediction on the three-dimensional pork model.

[0147] (5) Cutting trajectory extraction. Given a new three-dimensional pork model, a preset starting position and a cutting target, the model predicts and extracts an optimal cutting path according to the input parameters.

[0148] Cutting trajectory planning generation and its evolution model

[0149] Figure 4 The cutting trajectory planning model of the dual-arm robot pork cutting path autonomous planning method is shown. As shown in Figure 4 The cutting trajectory planning task includes an initial training phase and an evolution phase.

[0150] The data preprocessing module is to normalize the 3D model samples and corresponding trajectory label data in the large database for subsequent model training. First, the 3D model containing color information is down-sampled to reduce the amount of calculation, then the features are extracted, and a certain number of feature points are retained as a standard sample. The sample contains three-dimensional position coordinates and RGB channel color information, i.e. n 6-dimensional data. The label path data uses Huber regression straight line fitting method, considering the determination of the knife point, knife blade attitude and movement distance when cutting, finally converting the data into 7-dimensional format. In the initial training phase, after the data passes through the fully connected input layer, it is sequentially passed through several multi-layer network blocks composed of convolution, pooling, activation and normalization layers, and finally the calculation record is output by the output layer. The predicted path data of the output layer is calculated with the label path data for Loss calculation for gradient update, iterative training model. In the evolution training phase, the regularization term of the model Loss function is adjusted, and the feedback score part from the evolution module is added, so that the path trajectory generated by the model is more reasonable and effective in the actual operation of the robot. It should be noted that the two training phases of the network model are running on the algorithm server, and the model deployed on the video edge processor only contains the forward inference part.

[0151] In the modeling and training process of the cutting trajectory planning model, a reinforcement learning algorithm is used to optimize the meat cutting trajectory, and the optimization process includes the following steps:

[0152] Firstly, the robot needs to extract the shape, size, segmentation area and other state features of the meat block according to the environmental information of the 3D point cloud data and image data. The pre-processing stage converts these data into state information available for learning. When defining the state space and action space, the algorithm of reinforcement learning needs to clearly define the state that the robot can perceive and the action that it can perform. The state space includes the position of the robot cutter, the state of the cutting tool, the hardness and the cutting site; the action space includes adjusting the angle of the cutting tool, changing the cutting path, adjusting the cutting depth and speed, etc. The robot selects the best action from the current state through continuous exploration and utilization;

[0153] Secondly, by changing the selection of robot actions through the different force values of soft meat and bone feedback, changing the trajectory path through the error of the robot cutter and workbench position, determining whether the meat block is cut through, and the network result prediction of the force feedback value and position error value of each action of the robot state, then giving positive and negative rewards to each action, selecting the action with the maximum reward, that is, the optimal cutting path, to guide the robot to operate more accurately and efficiently. During the training process, the planned trajectory learns through continuous interaction with the cutting process, and the policy network is updated continuously, so that the robot can optimize the cutting path through continuous attempts, and gradually approach the optimal solution through multiple iterations.

[0154] Dual-arm robot control method based on machine vision and current feedback

[0155] Figure 5 The dual-arm robot control method is shown. Figure 2 The position of the operator is replaced by a manipulator control cabinet. Two manipulators, large and small, are installed on the control cabinet. The large manipulator performs left-hand functions, mainly including some auxiliary actions and heavy-load work; the small manipulator performs right-hand functions, mainly including complex cutting actions. The control program of the manipulator is implanted in the video edge processor, and the trajectory given by the "cutting trajectory planning model" is used for cutting. In the process of cutting, the current of each joint of the manipulator is calculated and fused with video data to obtain the force value during bone removal. Compared with the scheme using tactile sensors, it has the advantages of low cost and wide perception range.

[0156] Dual-arm robot co-evolution method

[0157] Figure 6 The dual-arm robot co-evolution method is shown. Figure 6 The video edge processor in Figure 5 only processes the video and point cloud data of the four cameras shown in Figure 6The illustrated multi-group dual-arm robot. Each dual-arm robot is equipped with a video information processor, which processes the four cameras equipped in the area where the dual-arm robot is located, and drives the two mechanical arms of the dual-arm robot to act. Figure 6 In the middle, the meat block moves from the first processor to the 3# processor. After the first dual-arm robot finishes processing the meat block, it can pass the processed information to the second processor as one of the path planning references. The above information is passed back one by one to realize the coordinated evolution of the subsequent mechanical arm path planning, thereby improving the accuracy of the meat cutting path.

[0158] The application is based on the mechanical arm cutting path autonomous evolution method of livestock carcass cutting big data and the dual-arm robot livestock carcass cutting system. Based on big data technology, point cloud data modeling technology, artificial intelligence technology, etc., a universal fine cutting process robot end running track planning and coordinated evolution method is proposed, which avoids the interference of factors such as size, placement position, and meat composition of meat products on the production line on the robot end running track planning, improves the planning accuracy of the robot end running track. The dual-arm robot with left and right hands completely simulates the operation mode of the operator, can adapt to the cutting process of any part, and at the same time reduces the development cost of multiple hardware. At the same time, the first and second industrial cameras record the operation process of the dual-arm robot, and combine with the artificial scoring of the cutting result to complete the trajectory planning evolution based on reinforcement learning, and achieve the optimal. The adjacent dual-arm robots on the assembly line communicate with each other to realize the information coordination of the evolution process.

Claims

1. A method for automatically evolving the cutting path of a mechanical arm based on large data of livestock carcass segmentation, comprising an initial training stage and a use evolution stage; characterized in that: the initial training stage, the process includes: step S1.1, install first and second RGBD cameras and first and second industrial cameras around the cutting operator, and input the on-site video signal into the video edge processor deployed on site; step S1.2, the video edge processor fuses the walking speed of the conveyor belt according to the 3D video information transmitted by the first and second RGBD cameras, and performs three-dimensional structure modeling on the meat blocks conveyed on the conveyor belt to obtain a 3D model with color texture characteristics, and uploads it to the video server; step S1.3, the video edge processor records the cutting process of the operator according to the 2D video information transmitted by the first and second industrial cameras and the 3D model with color texture characteristics established in step S1.2, and uploads the recorded operation process of the operator to the video server; step S1.4, an algorithm server is provided, which reads the previously stored data from the video server and completes the training of the related algorithm model: which includes: step S1.4-1, according to the 3D model with color texture characteristics of the meat block and the operation process video information of the operator, learn and extract the corresponding cutting path to obtain a cutting trajectory extraction model; the modeling process of the cutting trajectory extraction model includes the following five steps: (1) data preprocessing: first, divide each video by frame, extract the continuous action frame sequence, and use the labeling tool to label each frame; use the labeled data to train the network model to automatically detect and track the position of the cutter in each frame; then, detect multiple key points of the cutter, calculate the angle of the cutter relative to the horizontal line, and store the angle value calculated for each frame together with the position information; then, visualize the cutter trajectory and angle, check the continuity of the trajectory, identify and correct abnormal points, confirm the reasonableness of the cutting angle, and avoid extreme or impossible angle values; finally, divide the 3D model into grids, export the grid into the required format for subsequent physical simulation and spatial layout analysis; (2) feature extraction and action recognition: first, use a convolutional neural network to extract image features from each frame in the video to obtain the motion features and cutting angle information of the cutter; then, use a time series model to model the continuous action frames to capture the time sequence relationship of the action; then, the output of the time series model is classified by an action classification algorithm classifier to identify different cutting action patterns; (3) three-dimensional cutting process simulation: use a physics engine to simulate the interaction between the cutter and the 3D model to ensure that the cutting path conforms to the physical constraints; understand the position and angle changes of the cutter at different cutting stages through three-dimensional space analysis; use path planning algorithms to generate cutting paths in three-dimensional space and perform collision detection to ensure the feasibility and safety of the planned path; (4) model training and mapping relationship learning: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The supervised learning method is used to learn the mapping relationship between the cutting action and the cutting position based on the labeled cutting action and position data; the cutting mode learned from the video is migrated to the three-dimensional space through transfer learning, so as to ensure that the model can make accurate cutting prediction on the 3D model; (5) cutting trajectory extraction: Given a new 3D model, a new starting position and a cutting target, an optimal cutting path is predicted and extracted according to the input parameters; Step S1.4-2, the 3D model and the path information given by the cutting trajectory extraction model are used as a training sample set, and a cutting trajectory planning generation model is trained; The cutting trajectory extraction model and the cutting trajectory planning generation model are stored in the algorithm server; In the use evolution stage, the cutting trajectory planning generation model stored in the algorithm server is deployed on the video edge processor to plan the cutting path of the dual-arm robot for replacing the manual operation in the operation area, and to simulate the operation of the worker to complete the segmentation of the meat block.

2. The method for autonomous evolution of robotic arm segmentation paths based on large-scale animal carcass segmentation data according to claim 1, characterized in that: The process of the use evolution stage includes: First, the cutting trajectory planning generation model completes the planning of the left and right hand cutting paths of the dual-arm robot according to the 3D model of the meat block with color texture, including the running posture of the left and right hands of the robot and the running trajectory of the cutter; Then, the video edge processor sends the planned cutting path to the dual-arm robot through the interface, and the dual-arm robot segments the meat block according to the planned cutting path; During the cutting process according to the planned cutting path, the dual-arm robot performs closed-loop control on the cutting path according to the end cutting force feedback, and the end cutting force feedback is obtained by using the joint current estimation method. 3.The method of claim 1 or 2, wherein the method is characterized by: In step S1.4-2, the modeling process of the cutting trajectory planning generation model includes an initial training stage and a use evolution stage: Initial training stage: First, the 3D point cloud data containing color information of the 3D model is down-sampled to reduce the calculation amount, and then the feature extraction is performed, and a certain number of feature points are reserved as a group of standard samples; the sample contains three-dimensional position coordinates and RGB channel color information, i.e. n 6-dimensional data; the label path data adopts a regression straight line fitting method, which considers the determination of the knife point, the knife blade posture and the movement distance when the knife is lowered, and converts the data into a 7-dimensional format; In the initial training stage, after the data passes through the fully connected input layer, it is sequentially passed through a plurality of multi-layer network blocks composed of convolution, pooling, activation and normalization layers, and finally the calculation result is output by the output layer; the prediction path data of the output layer is calculated with the label path data for gradient update, and the model is iteratively trained; In the use evolution stage, the regularization term of the model Loss function is adjusted, and the feedback score part from the evolution module is added. 4.The method of claim 1 or 2, wherein the method is characterized by: An evolutionary module based on reinforcement learning is used to optimize the cutting trajectory planning generation model, video information of the operation process of the robot is forwarded to a video server by a video edge processor, a model in the algorithm server is used for optimization learning, the cutting trajectory planning generation model is evolved according to the operation video of the robot and the corresponding scoring condition, and the model optimized by the algorithm server is deployed to the video edge processor again at a suitable time, so that iteration of the whole control system is completed. 5.The method of claim 4, wherein the method further comprises: determining a path of the mechanical arm based on the big data of the livestock carcass dissection; and determining a dissection path of the mechanical arm based on the path of the mechanical arm. The cutting trajectory of the meat block is optimized by using a reinforcement learning algorithm: The robot needs to extract the shape, size and segmentation region state features of the meat block according to the environmental information of 3D point cloud data and image data, and the data is converted into state information available for learning in the preprocessing stage; The reinforcement learning algorithm defines the state that can be perceived by the robot and the action that can be executed by the robot, defines a state space and an action space, and the state space includes the position of the robot cutter, the state of the cutting tool, the hardness and the cutting position; The action space includes adjusting the angle of the cutting tool, changing the cutting path, adjusting the cutting depth and speed, and the robot selects the best action from the current state; By changing the selection of the robot action according to the different force values of the soft meat and bone feedback, the trajectory path is changed, and the position error of the robot cutter and the workbench is used to determine whether the meat block is cut, the network result prediction of the force feedback value and the position error of each action of the robot state is carried out, then positive and negative rewards are given to each action, and the action with the maximum reward, that is, the optimal cutting path, is selected; In the evolutionary process, the planned trajectory learns through continuous interaction with the cutting process, and the strategy network is updated continuously, so that the robot can optimize the cutting path through continuous attempts, and gradually approaches the optimal solution through multiple iterations. 6.The method of claim 1, 2 or 5, wherein: A double-arm robot cooperative evolution method is adopted, a plurality of double-arm robots are arranged in series on a flow line, each double-arm robot is provided with a video information processor, video signals of four cameras arranged in the area where the double-arm robot is located are processed, and two mechanical arms of the double-arm robot are driven to move; The meat block moves from the first processor to the rear one by one, after the first double-arm robot finishes processing the meat block, the processed information is transmitted to the second processor as a reference for path planning, the above information is transmitted level by level to the rear, the cooperative evolution of the path planning of the subsequent mechanical arm is realized, and the accuracy of the meat cutting path is improved.

7. A dual-arm robot animal carcass segmentation system based on the autonomous evolution method for robotic arm segmentation paths according to any one of claims 1-6, characterized in that: It comprises: A data acquisition device: A first RGBD camera and a second RGBD camera are arranged in front of and behind the left hand of the operator to record video information of the meat block passing through the conveying belt; A first industrial camera and a second industrial camera are arranged in front of and behind the right hand of the operator to record video information of the segmentation process of the operator; Video signals of the four cameras are input into a video edge processor arranged on site; A video edge processor and a video server: According to the 3D video information transmitted by the first RGBD camera and the second RGBD camera, the walking speed of the conveyor belt is fused, and the meat block conveyed on the conveyor belt is modeled in three-dimensional structure to form a three-dimensional structure model with color texture characteristics; According to the 2D video information transmitted by the first industrial camera and the second industrial camera, the segmentation process of the operator is recorded; The three-dimensional structure model of the meat block with color texture and the operation process of the operator are uploaded to the video server; The video edge processor is composed of an ARM processor and an FPGA chip; the ARM processor runs a Linux operating system to complete the scheduling of various functions; and the FPGA chip realizes the running of artificial intelligence algorithms; Algorithm server: Read the stored data from the video server to complete the training of the cutting trajectory extraction model and the cutting trajectory planning generation model related algorithm model; Double-arm robot: The algorithm server deploys the stored cutting trajectory planning generation model to the video edge processor; The video edge processor sends to the double-arm robot through the interface, and the double-arm robot simulates the operation mode of the operator to complete the segmentation of the meat block according to the planned trajectory.

8. The dual-arm robotic carcass splitting system of claim 7, wherein: The double-arm robot includes two mechanical hands, one large and one small, wherein the large mechanical hand completes the left hand function, including auxiliary action and load work; and the small mechanical hand completes the right hand function, including cutting action.

9. The dual-arm robotic carcass splitting system of claim 7 or 8, wherein: The video edge processor communicates with the video server and the algorithm server through wifi or industrial Ethernet; in the process of cutting and bone removal, the route is closed loop controlled according to the end cutting force feedback, wherein the end force feedback is realized by estimating the current of each joint of the mechanical hand and fusing the video data to calculate.

Images