AI-based mechanical arm sorting control method, system, device and medium

Abstract

The application relates to an AI-based mechanical arm sorting control method, system, device and medium. The method comprises the following steps: collecting a target squid RGB image and a depth image and performing standardization preprocessing to obtain a preprocessed image data pair; generating an instance segmentation mask, a visual classification confidence and three-dimensional point cloud data of the target squid through a multi-task deep learning model, and calculating the physical size features of each target squid, combining the visual classification confidence and the instance segmentation mask to generate a target structured list; generating a collision-free motion trajectory and an optimal grabbing point for each idle mechanical arm assigned target to generate a sorting instruction to instruct the mechanical arm assigned with the task to sort according to the collision-free motion trajectory, the optimal grabbing point and preset grabbing parameters. The method can effectively solve the problems of low efficiency, non-uniform classification standard and great influence of subjective factors in traditional manual sorting, and greatly improves the speed, accuracy and stability of squid sorting.

Description

Technical Field

[0001] This invention belongs to the field of fish catch sorting technology, and in particular relates to AI-based robotic arm sorting control methods, systems, equipment and media. Background Technology

[0002] With the continuous development of deep-sea fishing technology, especially the increased automation of squid fishing operations, the efficiency of onboard catch processing has become a key factor restricting the overall operation cycle and economic benefits. Against this backdrop, the core step in improving the quality and added value of the catch is to quickly and accurately grade and classify the squid of varying shapes and sizes, and promptly transfer them to refrigeration.

[0003] In traditional techniques, the sorting work described above is entirely manual. Workers must manually sort the squid into different sizes (large, medium, small) and place them into different collection boxes by sight and experience on a swaying deck. This process not only constitutes the main manual labor link in the "catch-process" chain, but its sorting speed and accuracy also depend heavily on the workers' skill level and physical condition.

[0004] However, the current manual sorting method has significant problems. First, the sorting efficiency is low and unstable, making it difficult to match the high catch rate of modern fishing machines. This easily leads to the accumulation of catch, prolonging the time window from deck to cold storage and affecting the freshness of the squid. Second, the standards for manual visual sorting are difficult to standardize and are easily affected by subjective factors such as fatigue and lighting, resulting in poor sorting accuracy, which is detrimental to subsequent refined processing and sales. Furthermore, the working environment is extreme, high-intensity, and monotonous. Facing the global labor shortage and aging population, the difficulty in recruiting workers and the continuous rise in labor costs have become serious challenges to the sustainable development of the industry. Summary of the Invention

[0005] Therefore, it is necessary to provide an intelligent sorting control method that can automate, accurately, continuously and stably complete squid sorting operations, thereby replacing manual labor and improving processing efficiency and quality consistency, in order to address the aforementioned technical problems.

[0006] Firstly, this application provides an AI-based robotic arm sorting control method, including:

[0007] In response to the target trigger signal within the conveyor belt identification area, RGB images and depth images of the squid target are acquired, and the RGB images and depth images of the squid target are standardized and preprocessed to obtain preprocessed image data pairs.

[0008] Based on preprocessed image data, input to a multi-task deep learning model to generate instance segmentation masks, visual classification confidence scores, and 3D point cloud data for squid targets;

[0009] Based on 3D point cloud data, the physical size features of each squid target are calculated, and a target structured list is generated based on the physical size features, visual classification confidence, and instance segmentation mask. The target structured list includes the 3D position, final classification label, and instance segmentation mask of each squid target.

[0010] Based on the target structured list, each idle robotic arm is assigned a target to be grasped, and motion trajectory planning is performed for each target to be grasped to generate a collision-free motion trajectory and the optimal grasping point.

[0011] Based on the collision-free motion trajectory and the optimal gripping point, sorting instructions are generated; among them, the gripping instructions are used to instruct the robotic arm assigned the task to sort the squid targets according to the collision-free motion trajectory, the optimal gripping point and the preset gripping parameters, and obtain the sorting results.

[0012] Furthermore, the method also includes:

[0013] After executing the grabbing command, the weight of the target collection box is obtained in real time, and it is determined whether the weight has reached the preset full load threshold, and the judgment result is obtained.

[0014] If the judgment result is that the weight has reached the preset full load threshold, a transfer instruction is generated. The instruction is used to instruct the target collection box to be transferred to the cold storage room and to replenish the empty collection box at the location of the target collection box.

[0015] Furthermore, based on the preprocessed image data, the input to the multi-task deep learning model generates instance segmentation masks, visual classification confidence scores, and 3D point cloud data for the squid target, including:

[0016] The preprocessed image data pair of squid target RGB image is input into the feature extraction backbone network of the multi-task deep learning model to extract multi-layer feature maps, and the multi-layer feature maps are input into the feature pyramid network of the multi-task deep learning model to generate fused feature maps.

[0017] The fused feature map is input into the parallel detection head of the multi-task deep learning model, and the bounding box, visual classification confidence vector and instance segmentation mask of each detected squid target are output.

[0018] For each squid target, based on the corresponding instance segmentation mask, extract all 3D points belonging to the squid target from the squid target depth image of the preprocessed image data pair, and construct the 3D point cloud data of the squid target based on all 3D points.

[0019] Furthermore, based on the 3D point cloud data, the physical size features of each squid target are calculated, and a structured list of targets is generated based on the physical size features, visual classification confidence, and instance segmentation mask. This structured list includes the 3D location, final classification label, and instance segmentation binary mask for each squid target, including:

[0020] Calculate the principal component orientation of the 3D point cloud data of each squid target, and construct the minimum axial bounding box of the squid target based on the principal component orientation;

[0021] The longest axis length of the minimum axial bounding box is determined as the physical size feature value of the squid target, and the physical size feature value of each squid target is mapped to a classification weight vector according to the preset size classification threshold.

[0022] The mean value of the three-dimensional point cloud data of each squid target is calculated to obtain the three-dimensional center position coordinates of the squid target;

[0023] Based on the visual classification confidence vector and classification weight vector of each squid target, the classification probability vector of the squid target is calculated using the following formula:

[0024]

[0025] in, Indicates the first The classification probability vector of each squid target. The preset fusion weight coefficients, The first output of the model Visual classification confidence vectors for individual squid targets Indicates by the first The classification weight vector for each squid target;

[0026] The category corresponding to the highest probability value in the classification probability vector is determined as the final classification label of the squid target. The three-dimensional center position, final classification label and instance segmentation mask of each squid target are aggregated to generate a structured list of targets.

[0027] Furthermore, based on the structured list of targets, each idle robotic arm is assigned a target to be grasped, and motion trajectory planning is performed for each target to be grasped, generating a collision-free motion trajectory and an optimal grasping point, including:

[0028] Iterate through the list of currently idle robotic arms and obtain the 3D position of the end effector of each idle robotic arm in the list.

[0029] The squid targets in the target structured list are identified as targets to be grabbed, and candidate grab points are obtained in the body region of the target to be grabbed based on the instance segmentation mask and 3D point cloud data of each target to be grabbed.

[0030] Calculate the crawling quality score for each candidate crawling point, and determine the crawling point with the highest crawling quality score for each target to be crawled as the optimal crawling point;

[0031] Based on the three-dimensional position of each end effector, calculate the Euclidean distance between the end position of each idle robotic arm and the optimal gripping point of all targets to be gripped, and generate a target-robotic arm binding list based on the Euclidean distance and preset binding requirements.

[0032] Based on the target-robotic arm binding list, motion trajectory planning is performed on each bound target-robotic arm pair to generate a collision-free motion trajectory from the current position of the idle robotic arm to the optimal grasping point of the target to be grasped.

[0033] Secondly, this application also provides an AI-based robotic arm sorting control system, including:

[0034] The image acquisition module is used to acquire RGB images and depth images of squid targets in response to target trigger signals within the recognition area of ​​the conveyor belt, and to perform normalization preprocessing on the RGB images and depth images of squid targets to obtain preprocessed image data pairs.

[0035] The data processing module is used to generate instance segmentation masks, visual classification confidence scores, and 3D point cloud data of squid targets by inputting preprocessed image data to a multi-task deep learning model.

[0036] The list building module is used to calculate the physical size features of each squid target based on 3D point cloud data, and generate a structured list of targets based on the physical size features, visual classification confidence and instance segmentation mask; wherein, the structured list of targets includes the 3D position of each squid target, the final classification label and the instance segmentation mask;

[0037] The trajectory acquisition module is used to assign targets to be grasped to each idle robotic arm based on a structured list of targets, and to plan the motion trajectory for each target to be grasped, generating a collision-free motion trajectory and the optimal grasping point;

[0038] The instruction generation module is used to generate sorting instructions based on the collision-free motion trajectory and the optimal gripping point. Among them, the gripping instruction is used to instruct the robotic arm assigned the task to sort the squid targets according to the collision-free motion trajectory, the optimal gripping point and the preset gripping parameters, so as to obtain the sorting results.

[0039] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement any of the personalized learning path generation methods based on learning behavior profiles described in the embodiments of this application.

[0040] Fourthly, this application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by a processor to implement the personalized learning path generation method based on learning behavior profiles as described in any of the embodiments of this application.

[0041] The aforementioned AI-based robotic arm sorting control method, system, equipment, and medium acquire RGB and depth images of squid targets and perform standardized preprocessing to obtain preprocessed image data pairs. Using a multi-task deep learning model, it generates instance segmentation masks, visual classification confidence scores, and 3D point cloud data for the squid targets, calculates the physical size features of each squid target, and combines the visual classification confidence scores and instance segmentation masks to generate a structured list of targets. It assigns targets to be grasped to each idle robotic arm and generates a collision-free motion trajectory and optimal grasping point for each target. This generates sorting instructions that direct the assigned robotic arms to sort the squid targets according to the collision-free motion trajectory, optimal grasping point, and preset grasping parameters. This effectively solves the problems of low efficiency, inconsistent classification standards, and significant susceptibility to subjective factors in traditional manual sorting, significantly improving the speed, accuracy, and stability of squid sorting while reducing labor costs. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating an AI-based robotic arm sorting control method in one embodiment.

[0044] Figure 2 This is a flowchart illustrating the steps of generating an instance segmentation mask, visual classification confidence score, and 3D point cloud data of a squid target based on preprocessed image data input to a multi-task deep learning model in one embodiment.

[0045] Figure 3 This is a schematic diagram of an AI-based robotic arm sorting control system in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] In one embodiment, an AI-based robotic arm sorting control method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through the interaction between the terminal and the server. Figure 1 As shown, in this embodiment, the method includes the following steps:

[0048] Step S101: In response to the target trigger signal within the conveyor belt identification area, acquire the squid target RGB image and squid target depth image, and perform normalization preprocessing on the squid target RGB image and squid target depth image to obtain preprocessed image data pairs.

[0049] Among them, a conveyor belt is a mechanical device that uses continuously moving conveying components (such as belts, rollers, chains, etc.) to realize the automated handling of materials or items, and is used to replace manual labor to complete point-to-point continuous conveying; a target trigger signal is a signal used to start or activate an action, process or response against a preset target in a specific scenario.

[0050] For example, when the conveyor belt transports the catch (squid) to the pre-calibrated identification area, precise pulse control commands are sent to the conveyor belt servo drive. These commands instruct the conveyor belt to stop immediately after the target has fully entered the center of the field of view, ensuring the target is in the optimal imaging position. Subsequently, spatiotemporally calibrated high-resolution RGB industrial cameras and depth cameras can acquire RGB and depth images of the squid target. Due to the variable lighting conditions on board and the presence of water vapor and salt spray interference, a dark channel prior-based dehazing algorithm can be applied to the original RGB image to restore clear image details. A limited-adaptive histogram equalization (CLAHE) algorithm can be used to enhance image contrast and highlight the target contour, resulting in a pre-processed RGB image. For the original depth image, median filtering can be used to remove outliers caused by surface reflections or noise, resulting in a pre-processed depth image. The preprocessed RGB image and the preprocessed depth image are strictly registered and aligned at the pixel level to form a one-to-one preprocessed image data pair. In this data pair, each valid spatial location simultaneously possesses color information (red, green, blue) and depth information (Z-axis).Among them, the conveyor belt servo driver is the core drive device controlling the operation of the conveyor belt servo motor. Essentially, it enables the conveyor belt to achieve "high-precision, high-response, stable and controllable" motion through precise signal processing and power output. Spatiotemporal calibration includes spatial calibration and temporal expression. Spatial calibration refers to eliminating lens distortion through algorithms and establishing the correspondence between image pixel coordinates and real physical coordinates (such as millimeters and micrometers), ensuring that the measured dimensions in the image (such as part length) are consistent with reality. Temporal calibration refers to synchronizing the camera's imaging time with the time axis of external equipment (such as a robotic arm) to avoid misalignment between the image and actual movement caused by imaging delay. High-resolution RGB industrial cameras focus light onto RGB sensors (red, green, and blue three-channel photosensitive elements) through a lens, converting them into digital signals to generate a color two-dimensional image including color and texture information. A depth camera is an imaging device that captures the distance (i.e., depth information) between objects in space and the camera. By acquiring the depth value of each pixel's distance from the camera's optical center, a depth image is formed. The dark channel prior-based dehazing algorithm reverse-engineers the degradation process of foggy images based on the general laws of naturally fog-free images (dark channel prior), thereby restoring a clear image. The contrast-limited adaptive histogram equalization algorithm is an improvement on the traditional adaptive histogram equalization algorithm. Its goal is to solve the problems of low local contrast and blurred details in images, while avoiding the over-enhancement and noise amplification defects that are common in traditional adaptive histogram equalization. Median filtering is a commonly used nonlinear signal (or image) denoising technique. It suppresses noise and preserves edge details by replacing the current value with the median value of pixels (or data points) within a local region. Strict registration and alignment is a core operation commonly used in many fields (such as image processing and data fusion). It enables high-precision spatial or logical matching and unification of multiple objects (such as images and data points) from different sources and potentially different forms. The pre-defined recognition region is a specific area with a limited recognition range, pre-defined manually or by an algorithm before performing the recognition task.

[0051] Step S102: Based on the preprocessed image data, input to the multi-task deep learning model to generate instance segmentation mask, visual classification confidence score, and 3D point cloud data of the squid target.

[0052] Among them, the multi-task deep learning model is a deep learning architecture that learns multiple related tasks at the same time. It improves the overall learning efficiency and performance by sharing information between tasks, rather than modeling each task separately as in the traditional single-task model.

[0053] For example, preprocessed image data is input into a pre-trained multi-task deep learning model for inference. This model uses an encoder-decoder architecture. The encoder part typically uses a convolutional neural network (CNN) as the backbone network to extract multi-level feature maps from the RGB image. These feature maps are then fed into a feature pyramid network (FPN) to fuse semantic and spatial detail information at different scales, generating a fused feature map rich in multi-scale contextual information. The decoder part of the model contains multiple parallel detection heads: one detection head is responsible for target detection and preliminary classification, outputting the bounding box and visual classification confidence vector for each identified squid target. This vector is used to represent the probability distribution of the target belonging to a predefined category (such as large, medium, and small); another detection head is responsible for instance segmentation, outputting a pixel-level binary mask for each detected target, i.e., an instance segmentation mask, which is used to accurately mark the pixel region occupied by the target in the image. Simultaneously, based on the registered depth information in the preprocessed data pairs, the two-dimensional coordinates (u, v) of each pixel and its corresponding depth value d are converted into three-dimensional spatial point coordinates (x, y, z) through the camera intrinsic parameter matrix. For each squid target, based on its instance segmentation mask, all three-dimensional points belonging to that target are selected from the entire scene's three-dimensional point set, thereby constructing the independent three-dimensional point cloud data corresponding to that target. Thus, three sets of key data are generated for each identified squid target: an instance segmentation mask that accurately delineates its shape, a visual classification confidence vector expressing its preliminary category judgment, and point cloud data describing its three-dimensional spatial morphology.Among them, convolutional neural networks (CNNs) are deep learning models inspired by biological visual systems. They can automatically extract spatial features and are particularly adept at processing data with grid structures, such as images and videos. Biological visual systems are complex functional systems that allow organisms to perceive external light signals, acquire spatial information, and form visual cognition. They are used to convert light stimuli from the physical world into neural signals that the brain can process, ultimately helping organisms achieve key survival behaviors such as localization, object recognition, and motion judgment. Feature pyramid networks are classic network structures in computer vision used for multi-scale object detection and recognition, addressing the performance bottleneck of traditional convolutional neural networks when processing objects of different sizes. Encoder-decoder is a classic sequence transformation architecture in deep learning. A pre-trained multi-task deep learning model is used to map an input sequence or data format to another output sequence or target format. A pre-trained multi-task deep learning model is a model that learns basic features (such as edge / texture patterns in images) on large-scale general data (such as massive amounts of unlabeled text) to form a general knowledge framework. The model incorporates adaptability design for multiple task scenarios during the pre-training stage, rather than optimizing only for a single task. It does not need to be trained from scratch. It can be quickly adapted to multiple specific tasks by fine-tuning with a small amount of target task data. The detection head is the core decision part of the model. It is responsible for transforming the general features extracted by the backbone network into specific detection task outputs, which directly determines whether the model can identify, locate, or segment the target.

[0054] Step S103: Based on the 3D point cloud data, calculate the physical size features of each squid target, and generate a target structured list based on the physical size features, visual classification confidence and instance segmentation mask; wherein, the target structured list includes the 3D position, final classification label and instance segmentation mask of each squid target.

[0055] For example, spatial geometric analysis is performed on the 3D point cloud data of each generated squid target to calculate its physical size features. Based on a pre-defined size classification threshold according to industry standards, such as a length threshold distinguishing between large, medium, and small, the calculated physical size feature values ​​are mapped to a size-based classification weight vector. This vector and the visual classification confidence vector have the same dimension (i.e., the number of categories). By weighted fusion of the classification weight vector and the visual classification confidence vector, a final classification probability vector for each squid target is obtained. The category with the highest probability value in the final classification probability vector is selected as the final classification label for that squid target. Simultaneously, the geometric center of the target's 3D point cloud data is calculated as its 3D position coordinates. The three core attributes—the 3D position coordinates, the final classification label, and the instance segmentation mask—of all squid targets are aggregated to generate a well-structured and information-complete target structured list. Among them, spatial geometric analysis is a method based on a three-dimensional spatial coordinate system, usually a rectangular coordinate system containing x, y, and z axes, to study the positional relationships, metric attributes, and morphological characteristics of geometric elements such as points, lines, surfaces, and volumes in space; size classification threshold is a critical numerical standard set to classify things according to size; mapping refers to assigning importance weights to different size features according to classification needs, and arranging the importance weights corresponding to each size feature in a fixed order into a vector form; weighted fusion is a data or information processing method that integrates the results of different sources and dimensions into a better comprehensive output based on their importance (i.e., weights).

[0056] Step S104: Based on the target structured list, assign targets to be grasped to each idle robotic arm, and perform motion trajectory planning for each target to be grasped to generate a collision-free motion trajectory and the optimal grasping point.

[0057] Among them, motion trajectory planning refers to planning a safe, efficient, and task-compliant motion path and time sequence for a moving subject (such as a robotic arm) from the initial state to the target state under given constraints.

[0058] For example, a real-time updated robotic arm status table is maintained. When task allocation is required, all robotic arms with a status of "idle" are selected from this table. The target structured list is traversed, and squid targets with a status of "to be grasped" are selected. For each target to be grasped, multiple candidate grasping points are sampled within the relatively solid body region indicated by its instance segmentation mask in the point cloud. For each candidate point, its grasping quality is comprehensively evaluated, considering factors such as whether the point is within the robotic arm's workspace (accessibility), the likelihood of the gripper approaching from a safe direction (accessibility), the stability against target slippage during grasping (anti-slip performance based on surface normals and friction coefficients), and whether the grasping action will interfere with the surrounding environment or other targets (obstacle avoidance). By quantifying and weighting these factors, the position with the highest comprehensive score for each target is selected as its optimal grasping point, and the 3D coordinates of this point and the recommended gripper approach direction are recorded. The Euclidean distance from the current 3D position of the end effector of each idle robotic arm to the optimal gripping point of all targets to be grasped is calculated. Based on "shortest distance priority" or other optimization strategies (such as global efficiency optimization), targets are assigned to the most suitable robotic arm, forming a temporary "target-robotic arm" binding relationship. For each bound "target-robotic arm" pair, a collision-free and kinematically feasible smooth trajectory is planned in real time, starting from the current joint state of the robotic arm and ending at a safe pre-grasping position above the optimal gripping point of the target. In biology / anatomy, the carcass region mainly refers to the core area of ​​the torso of an animal (including humans) after removing the head, limbs, and tail. It contains key body cavities such as the thoracic cavity, abdominal cavity, and pelvic cavity, as well as important internal organs such as the heart, lungs, liver, and gastrointestinal tract.

[0059] Step S105: Based on the collision-free motion trajectory and the optimal gripping point, a sorting instruction is generated; wherein, the gripping instruction is used to instruct the robotic arm assigned the task to sort the squid targets according to the collision-free motion trajectory, the optimal gripping point and the preset gripping parameters, and obtain the sorting result.

[0060] For example, the collision-free motion trajectory and optimal gripping point information generated for each target-robotic arm pair are encapsulated into specific sorting instructions that can be parsed by the actuator. This instruction is a structured data packet whose content may include: the trajectory sequence, the precise three-dimensional coordinates and orientation of the target's optimal gripping point, and preset gripping parameters retrieved based on the target's final classification label, such as the gripper's desired gripping force range and closing speed. This sorting instruction instructs the corresponding robotic arm controller to drive the servo motors of each joint of the robotic arm, ensuring it moves strictly according to the planned trajectory and smoothly to the pre-grip position near the optimal gripping point. Then, the controller switches to a force-position hybrid control mode, directing the end effector (such as an adaptive flexible gripper) to contact the target in a specified direction and adjust the gripping force in real time according to the gripping parameters in the instruction to achieve stable and compliant gripping, avoiding damage to the squid's surface. After successful gripping, the robotic arm controller transports the target above the collection box corresponding to the classification label and releases it, completing the subsequent transfer and placement actions. By monitoring the status feedback of the robotic arm controller, the completion of the sorting action is finally confirmed, and the target status and robotic arm status are updated. The trajectory sequence consists of a series of path points, each containing the target angle of each joint of the robotic arm or the pose of the end effector. Preset gripping parameters are core configuration items in data gripping (or information acquisition) scenarios; they refer to a set of key parameters pre-set before starting the gripping task to constrain and define gripping behavior, aiming to make the gripping process more accurate and efficient, avoiding invalid data or violations. A gripper is a mechanical device used to grip, fix, or transport objects. The robotic arm controller is the brain of the robotic arm system; its core function is to receive instructions, process information, and drive the robotic arm to complete precise movements, serving as the key hub connecting control requirements and mechanical execution. A robotic arm system is an automated equipment system that simulates human arm movements and can complete precise operations according to preset programs or real-time instructions, used to replace or assist humans in performing tasks such as gripping, transporting, assembling, and processing in specific scenarios. Encapsulation refers to transforming abstract sorting requirements into standardized instructions that the robotic arm controller can accurately recognize and execute.

[0061] In this embodiment, RGB and depth images of squid targets are acquired and standardized preprocessed to obtain preprocessed image data pairs. A multi-task deep learning model is used to generate instance segmentation masks, visual classification confidence scores, and 3D point cloud data for the squid targets. Based on the 3D point cloud data, the physical size features of each squid target are calculated, and a structured list of targets is generated by combining the visual classification confidence score and instance segmentation mask. Targets to be grasped are assigned to each idle robotic arm, and a collision-free motion trajectory and optimal grasping point are generated for each target. Based on the collision-free motion trajectory and optimal grasping point, sorting instructions are generated to instruct the assigned robotic arms to sort the squid targets according to the collision-free motion trajectory, optimal grasping point, and preset grasping parameters. This effectively solves the problems of low efficiency, inconsistent classification standards, and significant subjective influence in traditional manual sorting, greatly improving the speed, accuracy, and stability of squid sorting, reducing labor costs, and shortening the time window from squid on the deck to sorting completion, thus helping to ensure squid freshness and product added value.

[0062] In one exemplary embodiment, the method further includes:

[0063] Step S201: After executing the grabbing command, the weight of the target collection box is obtained in real time, and it is determined whether the weight has reached the preset full load threshold, and the judgment result is obtained.

[0064] The target collection box refers to the specific collection container that receives the squid being sorted when the grabbing instruction is executed; the preset full load threshold is a weight critical value that is pre-calculated and set based on the physical volume of the collection box, the average density of the squid, and the packing compactness required by the process.

[0065] For example, after confirming that the robotic arm has completed a squid grabbing and placement action and receiving a placement completion confirmation signal from the robotic arm controller, the current stable weight reading and its own ID of the target collection box are obtained. Then, in a preset collection box state mapping table, the corresponding record entry is indexed according to the ID. This entry pre-stores the category attribute of this collection box (corresponding to a large, medium, or small squid size) and its corresponding preset full-load threshold. The weight value will be obtained in real time. With the found threshold Perform a numerical comparison. If If so, a "not fully loaded" result is generated; if If the condition is met, a "full" result will be generated. The collection box status mapping table is essentially a structured data table or database table that specifically stores various status information of the "collection box", such as whether the collection box is enabled, the update time, etc. Each row is a record entry, corresponding to the complete status data of a collection box.

[0066] In step S202, if the judgment result is that the weight has reached the preset full load threshold, a transfer instruction is generated. The instruction is used to instruct the target collection box to be transferred to the cold storage room and to replenish the empty collection box at the location of the target collection box.

[0067] For example, when the judgment result is "full", a structured transfer instruction is generated. This instruction includes an action command, the target collection box ID (identification), and its current physical location code (such as the conveyor belt segment number). This instruction is sent to the programmable logic controller (PLC) that controls the conveyor belt or sliding mechanism at that location. The PLC drives the motor to start, smoothly moving the fully loaded collection box out of the main platform of the sorting work area and guiding it to the dedicated conveyor line leading to the ship's cold storage room. During the transfer process, radio frequency identification (RFID) readers installed at key nodes along the path automatically read the electronic tags on the collection boxes, verifying whether the ID matches the instruction, thus realizing verification and tracking of the transportation process. At the same time, the status of the fully loaded box is updated to "transporting" in the collection box status mapping table, and the physical workstation it originally occupied is marked as pending replenishment. A replenishment instruction is generated, which is used to instruct the crew to manually place an empty collection box of the same type to the vacant workstation or to instruct the automated storage and retrieval system or reserve area to automatically transport an empty box to the workstation. When a new empty collection box is in place, a new record is created in the status map table with that ID, the weight is initialized to zero, and the status is set to "empty and available," thus completing the resource reset of the entire workstation. Among these, a programmable logic controller (PLC) is a digital computing and operating electronic system designed specifically for industrial environments. It achieves automated control of industrial equipment or production processes through programming, replacing traditional relay control circuits. An electric motor (EMC) is a core power device that converts electrical energy into mechanical energy. An RFID reader is the core interactive device of an RFID system. Its main function is to wirelessly communicate with RFID tags (carriers storing target object information) via radio frequency signals, enabling the reading or writing of data within the tags, ultimately completing the identification, tracking, or information management of target objects. RFID systems are non-contact automatic identification technologies that achieve the identification and data interaction of target objects through radio frequency signals, without requiring physical contact or optical visibility. Automated storage and retrieval systems (AS / RS) are also included. A Retrieval System (AS / RS) is a modern warehousing solution that integrates automation technology, warehouse management systems, and automated racking systems. It achieves high efficiency and intelligence in the storage, retrieval, and management of goods by replacing manual labor with machines. A pre-planned transitional area (such as a pre-storage area in a warehouse) temporarily stores resources awaiting scheduling or use, preventing direct occupation of core operating areas (such as racks and production line workstations) and ensuring smooth processes. A warehouse management system is a digital tool used to optimize and manage the entire warehousing process, improving operational efficiency, reducing operating costs, and ensuring accurate inventory data.

[0068] In this embodiment, the weight of the collection box is acquired and determined in real time after each sorting action. Once a full load is detected, a series of logistics instructions, including transfer, verification, status update, and empty box replenishment, are automatically triggered and executed. This process seamlessly connects sorting operations with material flow, forming a complete "sorting-collection-transfer-resupply" closed loop. It avoids production interruptions or squid accumulation caused by untimely replacements in traditional manual inspection methods, greatly improves the continuous operation capability and overall throughput efficiency of the entire sorting line, and shortens the exposure time of squid in non-low-temperature environments, playing a key role in ensuring the initial freshness and quality of the product.

[0069] In one embodiment, such as Figure 2 As shown, based on preprocessed image data, the input to a multi-task deep learning model generates instance segmentation masks, visual classification confidence scores, and 3D point cloud data for squid targets, including:

[0070] Step S301: Input the RGB image of the squid target from the preprocessed image data pair into the feature extraction backbone network of the multi-task deep learning model to extract multi-layer feature maps, and input the multi-layer feature maps into the feature pyramid network of the multi-task deep learning model to generate fused feature maps.

[0071] Among them, the feature extraction backbone network is a core component in computer vision tasks (such as object detection), used to extract visual features from the original image layer by layer from low level to high level. This backbone network usually adopts a deep residual network (ResNet) or a similar convolutional neural network architecture with strong feature representation capabilities. This network is composed of a series of convolutional layers, pooling layers and activation function layers stacked together. Its working principle is to perform sliding calculations on the image through multiple layers of convolutional kernels, abstracting and extracting hierarchical information from low-level edges and textures to high-level semantic features layer by layer. The feature pyramid network fuses the strong semantic features of the high level with the precise localization features of the low level through a top-down path and lateral connections.

[0072] For example, the RGB image components of a preprocessed image data pair are input into the feature extraction backbone network of a pre-trained multi-task deep learning model. After the RGB image propagates forward through the entire backbone network, corresponding feature maps are extracted from specific deep, mid, and shallow layers of the network. These feature maps at different levels have different characteristics: deep feature maps have lower resolution but contain rich semantic information (e.g., "this is the squid's carcass"); shallow feature maps have higher resolution and can preserve the fine spatial details and contour information of the target (e.g., "this is the edge of the tentacle"). These multi-layered feature maps extracted from different levels are then fed into the model's feature pyramid network. The deepest semantic features are upsampled to increase their spatial size, and then fused with the feature maps from the corresponding layers of the backbone network through element-wise addition. This process is performed layer by layer, ultimately outputting a set of fused feature maps with the same number of channels but different spatial scales. Forward propagation is the core process of neural network computation in deep learning. Essentially, it is a forward flow that starts from the input data, passes information and performs calculations layer by layer, and finally obtains the prediction result. Upsampling is a commonly used technique in signal processing, image processing, and deep learning. Its core purpose is to improve the resolution or dimension of the data. Simply put, it makes sparse data denser and low-resolution content clearer (or larger in size). Element-wise addition refers to the operation of adding elements one by one according to their positions. The core requirement is that the two (or more) data structures (such as tensors) involved in the operation are completely matched in dimension / shape. This type of addition is only performed on elements at the same position, and the final output is a result with the same shape. Fusion is essentially the process by which different things break the original boundaries, penetrate each other, or combine, and finally form a new whole and generate new attributes.

[0073] Step S302: Input the fused feature map into the parallel detection head of the multi-task deep learning model, and output the bounding box, visual classification confidence vector and instance segmentation mask for each detected squid target.

[0074] For example, the generated multi-scale fused feature map is input into the head structure of a multi-task deep learning model, namely a set of parallel detection heads. One detection head is dedicated to object detection and classification. It can slide on the fused feature map according to a Region Proposal Network (RPN) or a similar mechanism to generate candidate regions (anchor boxes) that may contain the target, and then regress and classify these candidate regions. Finally, this detection head outputs a bounding box for each confirmed squid target. This box defines the minimum bounding rectangle of the target in the original image in pixel coordinates, and also outputs a visual classification confidence vector. This vector is a multi-dimensional array whose dimension is equal to the number of predefined categories (e.g., large, medium, and small). Each element in the vector represents the probability score of the target belonging to the corresponding category, and the sum of all elements is 1. An instance segmentation task is performed by another parallel detection head, whose structure may be a mask prediction branch. By receiving the same fused feature map and the Region of Interest (RoI) provided by the first detection head, a Fully Convolutional Network (FCN) performs binary classification on the pixels within each RoI (classifying them as belonging to the target or to the background), thereby generating an instance segmentation mask for each detected individual squid target. This mask is a binary matrix corresponding to the original image regions, where pixels with a value of "1" precisely mark all image regions occupied by that specific squid target. The Region Proposal Network (RPN) is a core component in object detection (especially in deep learning-based two-stage detection frameworks), used to efficiently generate candidate regions that may contain the target, replacing the time-consuming candidate region generation step in traditional methods, achieving end-to-end object detection. Regression refers to building a model to fit data patterns by analyzing the linear or non-linear relationship between independent variables (such as season) and dependent variables (such as temperature), ultimately outputting a continuous predicted value. Classification refers to dividing data into preset categories by learning data features (such as pixel features of images), outputting the probability of a sample belonging to a certain category or a direct probability. Category labels; The mask prediction branch is a sub-network in deep learning models (especially in the field of computer vision) specifically designed to complete the mask prediction task. It is usually a key component of complex models (such as two-stage / multi-stage detection models). Its core objective is to output a mask, binary or probability map, that accurately corresponds to the spatial location of the target object, used to label the pixel regions of the object; A fully convolutional network is a deep learning model composed entirely of convolutional layers (including convolution, pooling, deconvolution, etc.). Its core feature is that it abandons fully connected layers and uses convolution to achieve end-to-end dense prediction. It is mainly used for pixel-level tasks in image segmentation and object detection.

[0075] Step S303: For each squid target, based on the corresponding instance segmentation mask, extract all three-dimensional points belonging to the squid target from the squid target depth image of the preprocessed image data pair, and construct the three-dimensional point cloud data of the squid target based on all three-dimensional points.

[0076] In this preprocessed image data pair, the RGB image and the depth image have been strictly registered and aligned, which means that each image pixel position (u,v) has a completely consistent spatial correspondence in the RGB image and the depth image.

[0077] For example, for each squid target instance, the set of pixel coordinates with a value of "1" in the segmentation mask of that target instance is located. Based on these identical coordinates (u,v), the depth value d (i.e., the physical distance of the point from the camera) at each corresponding pixel position is found and read from the registered preprocessed depth image. Combining the pre-set camera intrinsic parameter matrix and the transformation relationship from the camera coordinate system to the world coordinate system, each two-dimensional pixel point (u,v) and its depth value d are converted into a point coordinate (x, y, z) in the three-dimensional world coordinate system through the principle of inverse perspective projection transformation. By traversing all foreground pixels of the target mask and repeating the above coordinate transformation, all three-dimensional spatial points belonging to the specific squid target are collected. The set of these three-dimensional points constitutes the three-dimensional point cloud data corresponding to the squid target. This point cloud data describes the surface geometry and spatial distribution of the squid target in three-dimensional physical space in the form of a discrete point set, and each point contains three-dimensional position information. Among them, the camera intrinsic parameter matrix is ​​the core parameter matrix describing the camera's own optical and geometric characteristics. It is used to establish the mathematical relationship between the pixel coordinates on the camera's imaging plane and the projection of 3D points in the camera coordinate system onto the 2D image plane. It is a key bridge from the 3D space of the camera's perspective to the 2D image. The world coordinate system is a globally defined reference coordinate system, while the camera coordinate system is a local coordinate system with the camera's optical center as the origin and the optical axis as the Z-axis. The transformation relationship between the two is essentially to describe the camera's position (translation) and orientation (rotation) in the world coordinate system using rigid body transformation, ensuring that the local 3D information seen by the camera can correspond to the global world coordinates. The inverse perspective projection transformation refers to the reverse derivation of the coordinates of the original point in 3D space from the projected coordinates of the 2D image plane. Its principle revolves around the inverse process of the forward perspective projection transformation.

[0078] In this embodiment, robust multi-scale visual features are extracted and fused through a backbone network and a feature pyramid network. A parallel detection head synchronously outputs the target's category probability, location boundary, and pixel-level contour. Based on the registered depth information, the two-dimensional contour is accurately mapped to a three-dimensional point cloud. This significantly improves the accuracy and efficiency of target detection and segmentation, and provides accurate and comprehensive target data for subsequent processes, effectively supporting the automated and intelligent operation of robotic arm sorting.

[0079] In one embodiment, based on 3D point cloud data, the physical size features of each squid target are calculated, and a target structured list is generated based on the physical size features, visual classification confidence, and instance segmentation mask; wherein, the target structured list includes the 3D position of each squid target, the final classification label, and the instance segmentation binary mask, including:

[0080] Step S401: Calculate the principal component orientation of the 3D point cloud data of each squid target, and construct the minimum axial bounding box of the squid target based on the principal component orientation.

[0081] For example, Principal Component Analysis (PCA) is performed on the 3D point cloud dataset of each target to calculate the eigenvalues ​​of the matrix. The eigenvalues ​​represent the dispersion of the data along the corresponding eigenvector directions, while the eigenvectors indicate these directions. For example, the three eigenvectors with the largest eigenvalues ​​are selected, constituting the three principal component directions of the point cloud in space. These three directions are orthogonal to each other and jointly define a new local coordinate system, the origin of which is usually set at the centroid of the point cloud. Based on these three principal component directions, the minimum axial bounding box (OBB) of the squid target is constructed: the point cloud coordinates are transformed to this local coordinate system, and the maximum and minimum values ​​of the point cloud along the three principal axes are found to determine the size (length, width, height) and orientation of the bounding box. The parameters of the minimum axial bounding box of the target are obtained, including its center position, the vector directions of the three axes, and the half-side length along each axis. PCA is a statistical method that transforms a set of potentially correlated variables into a set of linearly uncorrelated variables (i.e., principal components) through orthogonal transformation. The first principal component has the largest variance (i.e., the data is most dispersed in that direction), the second principal component has the second largest variance in a plane orthogonal to the first principal component, and so on. An OBB is a minimal cuboid whose sides are parallel to the directions of the aforementioned principal components (i.e., local coordinate axes) and can tightly enclose all three-dimensional points. The covariance matrix is ​​a matrix describing the linear relationship between random variables and its own dispersion, used to transform multiple random variables... The covariance and variance of quantities are systematically organized to facilitate the analysis of the overall distribution characteristics of multivariate data; eigenvalue decomposition is an important decomposition method in linear algebra for square matrices (matrices with the number of rows equal to the number of columns). Its core is to decompose the square matrix into a form related to eigenvalues ​​and eigenvectors. In essence, it simplifies the analysis of linear transformations through the inherent properties of matrices; coordinate transformation is a fundamental tool in geometry and algebra. Its core is to transform the coordinate representation of the same point / figure between different coordinate systems. In essence, it describes the switching of perspectives without changing the position or shape of the point or figure itself, only changing the reference system used to locate them.

[0082] Step S402: The longest axis length of the minimum axial bounding box is determined as the physical size feature value of the squid target, and the physical size feature value of each squid target is mapped to a classification weight vector according to the preset size classification threshold.

[0083] The preset size classification threshold refers to the size classification threshold pre-defined according to industry sorting standards. This threshold defines the length boundary that distinguishes between large, medium, and small categories.

[0084] For example, after obtaining the minimum axial bounding box parameter for each squid target, the length of the bounding box along its three principal axes is extracted. By comparing these three length values, the maximum value is determined as the physical size feature value of the squid target. This physical size feature value is then mapped into a classification weight vector according to a preset size classification threshold. The mapping rules can be hard-score, for example, if the size feature value falls within the "middle" class interval, the weight vector is set to 1 in the "middle" class dimension and 0 elsewhere; or soft-score, for example, a smoothing function is calculated based on the distance to each threshold boundary, so that the sum of the values ​​of each dimension of the weight vector is 1, to express a certain degree of uncertainty. The principle for determining the physical size feature value is that its longest axis length usually best represents the overall size of a squid-like, approximately long and flexible object, such as the main length of its carcass. This physical size feature value is an objective scalar value based on three-dimensional geometric measurement, used to reflect the actual physical size of the target, and is not affected by perspective, occlusion, or lighting changes in two-dimensional images. The smoothing function calculation uses mathematical methods to filter noise and fluctuations in data or functions, generating smoother and more continuous results, thereby highlighting the overall trend of the data or the core features of the function.

[0085] Step S403: Calculate the mean value of the three-dimensional point cloud data of each squid target to obtain the three-dimensional center position coordinates of the squid target.

[0086] For example, to determine the reference position of each squid target in the 3D world, it is necessary to calculate the spatial center of its point cloud. The 3D coordinates (x, y, x) of all the 3D points belonging to that target (assuming there are N points) are then calculated. i ,y i ,z i Summing is performed separately for each coordinate axis (X, Y, Z). The sum on each axis is divided by the total number of points N to obtain the average value along the three coordinate axes. The calculation formula can be expressed as: The Y and Z coordinates are calculated similarly. This is how the coordinates of the three-dimensional center position are calculated. This coordinate represents the average spatial location of the target point cloud set and is a stable location estimate.

[0087] Step S404: Based on the visual classification confidence vector and classification weight vector of each squid target, the classification probability vector of the squid target is calculated using the following formula:

[0088]

[0089] in, Indicates the first The classification probability vector of each squid target. The preset fusion weight coefficients, The first output of the model Visual classification confidence vectors for individual squid targets Indicates by the first The classification weight vector of each squid target.

[0090] The preset fusion weight coefficient α (0≤α≤1) is used to balance the contributions of the two types of information. The value of α can be set through experiments or experience. For example, when it is believed that visual information is more reliable in most cases, α can be set to be greater than 0.5. softmax is an activation function used to transform an arbitrary real number vector into a probability distribution vector, such that each element of the output vector is between 0 and 1, and the sum of all elements is 1, thereby normalizing the original confidence score into a standardized probability form.

[0091] For example, based on the visual classification confidence vector and classification weight vector of each squid target, the formula is used to... The classification probability vector for the squid target is calculated. This formula combines a normalized visual probability distribution with a size-based weighted distribution in a linear weighted manner to generate the final comprehensive classification probability vector. .

[0092] Step S405: Determine the category corresponding to the highest probability value in the classification probability vector as the final classification label of the squid target, and aggregate the three-dimensional center position, final classification label and instance segmentation mask of each squid target to generate a structured list of targets.

[0093] For example, in obtaining the final classification probability vector for each squid target Then, iterate through all dimensions of the probability vector (i.e., the probability values ​​corresponding to each candidate category), and find the largest probability value through a simple comparison operation. The category index corresponding to this maximum value is determined as the final classification label for the squid target. For example, if the category order is [large, medium, small], and the probability vector has the largest value corresponding to the "medium" category, then the final classification label is "medium". This decision-making process is deterministic. For each squid target identified in the current field of view, three core attributes are extracted from its internal data: the three-dimensional center position coordinates, the newly determined final classification label, and the obtained instance segmentation mask. These attributes are organized according to the target index (e.g., j) to form a record array or list, i.e., a target structured list. Each record in the list uniquely and completely describes the key information of a target to be sorted: where it is (three-dimensional position), what it is (final category), and what its precise shape is (two-dimensional mask).

[0094] In this embodiment, principal component analysis and minimum bounding box construction are used to extract the longest axis feature sensitive to object size from the point cloud, and quantify it into an objective physical size feature value. According to industry standards, this feature value is mapped to a weight vector of the same dimension as the visual classification. After calculating the spatial location of the target, the visual classification probability based on appearance and the weight based on physical size are adaptively weighted to obtain the final classification probability. The category label is determined by maximizing the decision and aggregated with location and mask information to form a structured list. This effectively solves the problem of insufficient accuracy in single-feature classification, significantly improves the accuracy and reliability of squid target classification, and enhances sorting efficiency and quality consistency.

[0095] In one embodiment, based on a structured list of targets, targets to be grasped are assigned to each idle robotic arm, and motion trajectory planning is performed for each target to be grasped to generate a collision-free motion trajectory and an optimal grasping point, including:

[0096] Step S501: Traverse the list of robotic arms that are currently idle, and obtain the three-dimensional position of the end effector of each idle robotic arm in the list.

[0097] The list of robotic arms is maintained internally and updated dynamically. This list records the real-time working status of each robotic arm, such as "busy", "idle" or "faulty".

[0098] For example, when a new round of task allocation is needed, the list of robotic arms is traversed, and all robotic arms marked as "idle" are selected to form a list of currently available idle robotic arms. The precise 3D position of the end effector of each idle robotic arm in the list in a unified world coordinate system is obtained in real time. This position data is typically calculated by the robotic arm controller using its internal forward kinematics model. This model calculates the position and orientation of the end effector relative to the robotic arm base coordinate system based on feedback from the angle encoders of each joint. Then, it is transformed into world coordinates using a pre-calibrated transformation matrix from the base coordinate system to the world coordinate system. Among them, the end effector is the core component of a robot system that directly contacts the work object and performs specific tasks, equivalent to the robot's hand. Its design and function directly determine the types of tasks the robot can perform. The forward kinematics model (FK) is a core foundational model in fields such as robotics, computer animation, and mechanical design. It is used to calculate the pose (position + attitude) of the end effector (or key component) based on the joint motion parameters of a mechanical system (such as a robot arm). An angle encoder is a precision sensor that converts mechanical rotation angle or position information into electrical signals (such as digital signals) to achieve accurate mapping between mechanical angles and electronic data. A robot system is a comprehensive system that integrates multiple disciplines and can achieve autonomous or semi-autonomous operation, used to replace or assist humans in completing specific tasks.

[0099] Step S502: The squid target in the target structured list is identified as the target to be grabbed, and candidate grab points are obtained in the body region of the target to be grabbed based on the instance segmentation mask and three-dimensional point cloud data of each target to be grabbed.

[0100] For example, a structured list of targets is traversed, where each entry represents a squid target that has been identified and classified, but is initially in a "pending" state. All squid targets in the structured list are identified as targets to be grasped, forming a set of squid targets to be grasped. For each target in the set, a suitable grasping point is planned. Considering the biological structure of squid, its mantle is relatively solid and regularly shaped, making it an ideal area for stable grasping. Based on the instance segmentation mask attached to the target, the approximate area of ​​the mantle is located in the two-dimensional image domain, which can be achieved through simple algorithms such as convex hull analysis of the mask shape. Combining the three-dimensional point cloud data precisely registered with the mask, the two-dimensional mantle region is mapped to three-dimensional space to obtain a subset of three-dimensional points belonging to the surface of the target mantle. In this set of three-dimensional surface points, a spatial sampling algorithm can be used to generate a set of candidate three-dimensional grasping points. Each candidate point should include not only its 3D coordinates but also an estimated surface normal vector at that point. This normal vector estimate can be obtained by fitting the point cloud in the neighborhood of that point. The normal vector is crucial for determining the approach direction of the gripper. Convex hull analysis is a fundamental technique in computational geometry and data mining. Its core is to find the smallest convex polygon (2D scene) or convex polyhedron (3D scene) that can enclose all points from a discrete set of points; this smallest convex structure is the convex hull. Spatial sampling algorithms are a type of technique that selects discrete sample points from a continuous spatial region according to specific rules. The core goal is to efficiently represent the overall characteristics of space (such as geometry) with a limited number of samples, balancing sampling accuracy and computation / storage costs. Fitting refers to finding a mathematical function (or curve, surface) that closely approximates a set of known data points, thereby using this function to describe the underlying patterns in the data, fill in missing information, or predict unknown situations.

[0101] Step S503: Calculate the crawling quality score for each candidate crawling point, and determine the crawling point with the highest crawling quality score for each target to be crawled as the optimal crawling point.

[0102] For example, a grasping quality scoring function is defined, which comprehensively evaluates several key factors: 1) Accessibility, i.e., whether the point is within the workspace of at least one robotic arm; 2) Anti-slip performance, which depends on the angle between the gripper's preset gripping force direction and the normal vector of the grasping point surface; the smaller the angle, the larger the normal force component and the stronger the anti-slip capability; 3) Stability, evaluating the expected displacement of the target's center of mass after grasping the point; the smaller the displacement, the more stable the posture; 4) Obstacle avoidance, evaluating whether the path from the robotic arm approaching the point is prone to interference with other parts of the target (such as the wrist) or the surrounding environment. Sub-scoring functions are designed for each factor and normalized. Then, a weighted summation is used to calculate the comprehensive grasping quality score Q for each candidate grasping point, which can be expressed as: ,in, For accessibility scoring, For anti-slip rating, For stability rating, To prevent collision scoring, , , and These are the weights corresponding to the scores. All candidate points for a target are traversed, and their respective Q-values ​​are calculated. Finally, through simple comparison and sorting, the point with the highest grab quality score Q-value among all candidate points for the target is determined as the optimal grab point for the squid target, and its 3D coordinates and normal vector are recorded.

[0103] Step S504: Based on the three-dimensional position of each end effector, calculate the Euclidean distance between the end position of each idle robotic arm and the optimal gripping point of all targets to be gripped, and generate a target-robotic arm binding list based on the Euclidean distance and preset binding requirements.

[0104] For example, for each idle robotic arm i, the three-dimensional straight-line distance between its end effector position and the optimal gripping point position of each target j to be gripped is calculated. All The distances are aggregated to form a distance matrix, which visually reflects the spatial cost of each robotic arm reaching each target point. This is achieved by continuously searching for all possible distances. The minimum value in the list is used to assign target j to robotic arm i. Then, both the target and robotic arm are removed from the assignment list. This process is repeated until no robotic arm or target is available for assignment. Pre-defined constraints must also be considered during the assignment process, such as assigning only one target to each robotic arm at a time. The assignment results are recorded as a "target-robotic arm binding list," where each record specifies the correspondence between a target ID and the robotic arm ID to which it is assigned. Pre-defined constraints refer to pre-set restrictive rules or preconditions on the target range, variable conditions, or operational boundaries.

[0105] Step S505: Based on the target-robotic arm binding list, perform motion trajectory planning for each bound target-robotic arm pair to generate a collision-free motion trajectory from the current position of the idle robotic arm to the optimal grasping point of the target to be grasped.

[0106] For example, based on the generated binding list, independent motion trajectory planning is performed for each bound "target-robotic arm" pair. The goal of the planning is to generate a spatial path from the current end-effector position (known) of the robotic arm to a safe pre-grasp position above the optimal gripping point of the target, and this path must be collision-free. An integrated motion planner is invoked, which is typically based on a sampling algorithm, such as Rapidly-exploring RandomTree (RRT). The planner randomly samples in the configuration space of the robotic arm to construct a tree that grows from the starting state (current joint angle) to the target state (one of the inverse kinematic solutions corresponding to the gripping point). Throughout the planning process, a collision detection model provides real-time collision detection support to the planner. Each time a tree node is expanded, the planner performs a collision query to ensure that the newly generated path segment does not intersect with any obstacles. Finally, once the planner successfully finds a collision-free path connecting the start and end points, it performs post-processing on this path (such as smoothing and adding time parameters) to generate a collision-free motion trajectory consisting of a series of path points that can be directly executed by the robotic arm controller. The collision detection model includes not only the simplified geometry of the robotic arm's links (such as a cylindrical or cuboid bounding box) and the end effector model, but also the known static environment (such as a conveyor belt) and the real-time motion states of other robotic arms. Its position can be predicted through its planned or currently executing trajectory. The integrated motion planner is a core component responsible for coordinating path planning and motion control in autonomous mobile systems such as robots, autonomous vehicles, and drones. Its core objective is to ensure that the autonomous mobile system moves correctly (avoiding obstacles and meeting task objectives) and stably (meeting dynamic constraints and ensuring smooth and safe movements) in complex environments. Sampling algorithms are methods that extract a subset of samples from a target dataset or probability distribution, using fewer... Replacing complete data with representative samples solves analysis and computation problems when the data volume is too large, the distribution is complex, or the complete set cannot be directly obtained; Fast expanding random trees are a classic sampling algorithm in motion planning, aiming to efficiently explore unknown regions and find feasible paths from the starting point to the target point in complex environments (such as spaces with obstacles), especially suitable for high-dimensional spaces (such as robot joint space); Configuration space is an abstract mathematical space in physics, mathematics, and robotics that describes all possible states of a system; Random sampling is a core method in statistics for selecting a subset of individuals (i.e., samples) from a population of research objects. The core principle is that each individual in the population has an equal and independent chance of being selected into the sample, avoiding bias caused by human selection.

[0107] In this embodiment, by acquiring precise geometric information of idle robotic arm resources and targets to be processed, the optimal gripping point is selected for each target based on gripping quality assessment. Efficient dynamic matching between multiple robotic arms and multiple targets is achieved by calculating spatial distances and applying allocation strategies. A safe motion trajectory is planned in real time for each pair of matching relationships under complex constraints. This significantly improves the automation level and execution efficiency of robotic arm sorting operations, effectively replacing manual sorting and solving the problems of low efficiency and poor accuracy in traditional manual sorting.

[0108] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0109] Based on the same inventive concept, this application also provides an AI-based robotic arm sorting control system for implementing the AI-based robotic arm sorting control method described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more AI-based robotic arm sorting control system embodiments provided below can be found in the limitations of the AI-based robotic arm sorting control method described above, and will not be repeated here.

[0110] In one exemplary embodiment, such as Figure 3 As shown, an AI-based robotic arm sorting control system 300 is provided, including:

[0111] The image acquisition module 301 is used to acquire RGB images and depth images of squid targets in response to a target trigger signal within the recognition area of ​​the conveyor belt, and to perform normalization preprocessing on the RGB images and depth images of squid targets to obtain preprocessed image data pairs.

[0112] Data processing module 302 is used to generate instance segmentation mask, visual classification confidence and 3D point cloud data of squid target based on preprocessed image data input to a multi-task deep learning model;

[0113] The list construction module 303 is used to calculate the physical size features of each squid target based on 3D point cloud data, and generate a structured list of targets based on the physical size features, visual classification confidence and instance segmentation mask; wherein, the structured list of targets includes the 3D position, final classification label and instance segmentation mask of each squid target;

[0114] The trajectory acquisition module 304 is used to assign targets to be grasped to each idle robotic arm based on a target structured list, and to plan the motion trajectory for each target to be grasped, generating a collision-free motion trajectory and the optimal grasping point.

[0115] The instruction generation module 305 is used to generate sorting instructions based on the collision-free motion trajectory and the optimal gripping point; wherein, the gripping instruction is used to instruct the robotic arm assigned the task to sort the squid targets according to the collision-free motion trajectory, the optimal gripping point and the preset gripping parameters, and obtain the sorting results.

[0116] In one exemplary embodiment, the system further includes:

[0117] The weight judgment module is used to obtain the weight of the target collection box in real time after the grab command is executed, and to determine whether the weight has reached the preset full load threshold, and obtain the judgment result.

[0118] The collection box transfer module is used to generate a transfer instruction if the judgment result is that the weight has reached the preset full load threshold. The instruction is used to instruct the target collection box to be transferred to the cold storage compartment and to replenish the empty collection box at the location of the target collection box.

[0119] In one embodiment, the data processing module 302 is further configured to:

[0120] The preprocessed image data pair of squid target RGB image is input into the feature extraction backbone network of the multi-task deep learning model to extract multi-layer feature maps, and the multi-layer feature maps are input into the feature pyramid network of the multi-task deep learning model to generate fused feature maps.

[0121] The fused feature map is input into the parallel detection head of the multi-task deep learning model, and the bounding box, visual classification confidence vector and instance segmentation mask of each detected squid target are output.

[0122] For each squid target, based on the corresponding instance segmentation mask, extract all 3D points belonging to the squid target from the squid target depth image of the preprocessed image data pair, and construct the 3D point cloud data of the squid target based on all 3D points.

[0123] In one embodiment, the list building module 303 is further configured to:

[0124] Calculate the principal component orientation of the 3D point cloud data of each squid target, and construct the minimum axial bounding box of the squid target based on the principal component orientation;

[0125] The longest axis length of the minimum axial bounding box is determined as the physical size feature value of the squid target, and the physical size feature value of each squid target is mapped to a classification weight vector according to the preset size classification threshold.

[0126] The mean value of the three-dimensional point cloud data of each squid target is calculated to obtain the three-dimensional center position coordinates of the squid target;

[0127] Based on the visual classification confidence vector and classification weight vector of each squid target, the classification probability vector of the squid target is calculated using the following formula:

[0128]

[0129] in, Indicates the first The classification probability vector of each squid target. The preset fusion weight coefficients, The first output of the model Visual classification confidence vectors for individual squid targets Indicates by the first The classification weight vector for each squid target;

[0130] The category corresponding to the highest probability value in the classification probability vector is determined as the final classification label of the squid target. The three-dimensional center position, final classification label and instance segmentation mask of each squid target are aggregated to generate a structured list of targets.

[0131] In one embodiment, the trajectory acquisition module 304 is further configured to:

[0132] Iterate through the list of currently idle robotic arms and obtain the 3D position of the end effector of each idle robotic arm in the list.

[0133] The squid targets in the target structured list are identified as targets to be grabbed, and candidate grab points are obtained in the body region of the target to be grabbed based on the instance segmentation mask and 3D point cloud data of each target to be grabbed.

[0134] Calculate the crawling quality score for each candidate crawling point, and determine the crawling point with the highest crawling quality score for each target to be crawled as the optimal crawling point;

[0135] Based on the three-dimensional position of each end effector, calculate the Euclidean distance between the end position of each idle robotic arm and the optimal gripping point of all targets to be gripped, and generate a target-robotic arm binding list based on the Euclidean distance and preset binding requirements.

[0136] Based on the target-robotic arm binding list, motion trajectory planning is performed on each bound target-robotic arm pair to generate a collision-free motion trajectory from the current position of the idle robotic arm to the optimal grasping point of the target to be grasped.

[0137] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the AI-based robotic arm sorting control method as described above.

[0138] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0139] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0140] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. An AI-based robotic arm sorting control method, characterized in that, The method includes: In response to a target trigger signal within the conveyor belt identification area, an RGB image and a depth image of the squid target are acquired, and the RGB image and the depth image of the squid target are standardized and preprocessed to obtain preprocessed image data pairs. Based on the preprocessed image data, input to a multi-task deep learning model to generate instance segmentation masks, visual classification confidence scores, and 3D point cloud data of the squid target; Based on the 3D point cloud data, the physical size features of each squid target are calculated, and a target structured list is generated based on the physical size features, the visual classification confidence score, and the instance segmentation mask; wherein, the target structured list includes the 3D position, final classification label, and instance segmentation mask of each squid target; Based on the target structured list, each idle robotic arm is assigned a target to be grasped, and motion trajectory planning is performed for each target to be grasped to generate a collision-free motion trajectory and the optimal grasping point. Based on the collision-free motion trajectory and the optimal gripping point, a sorting instruction is generated; wherein, the gripping instruction is used to instruct the robotic arm assigned the task to sort the squid target according to the collision-free motion trajectory, the optimal gripping point and the preset gripping parameters, so as to obtain the sorting result.

2. The method according to claim 1, characterized in that, The method further includes: After executing the grabbing command, the weight of the target collection box is obtained in real time, and it is determined whether the weight has reached the preset full load threshold, and the determination result is obtained. If the judgment result is that the weight reaches the preset full load threshold, a transfer instruction is generated. The instruction is used to instruct the target collection box to be transferred to the cold storage room and to replenish the empty collection box at the location of the target collection box.

3. The method according to claim 1, characterized in that, The step of inputting the preprocessed image data into a multi-task deep learning model to generate instance segmentation masks, visual classification confidence scores, and 3D point cloud data for the squid target includes: The squid target RGB image of the preprocessed image data pair is input into the feature extraction backbone network of the multi-task deep learning model to extract multi-layer feature maps, and the multi-layer feature maps are input into the feature pyramid network of the multi-task deep learning model to generate fused feature maps. The fused feature map is input into the parallel detection head of the multi-task deep learning model, and the bounding box, visual classification confidence vector and instance segmentation mask of each detected squid target are output. For each squid target, based on the corresponding instance segmentation mask, all three-dimensional points belonging to the squid target are extracted from the squid target depth image of the preprocessed image data pair, and three-dimensional point cloud data of the squid target is constructed based on all the three-dimensional points.

4. The method according to claim 1, characterized in that, Based on the 3D point cloud data, the physical size features of each squid target are calculated, and a target structured list is generated based on the physical size features, the visual classification confidence score, and the instance segmentation mask; wherein, the target structured list includes the 3D position, final classification label, and instance segmentation binary mask of each squid target, including: Calculate the principal component orientation of the three-dimensional point cloud data for each squid target, and construct the minimum axial bounding box of the squid target based on the principal component orientation; The longest axis length of the minimum axial bounding box is determined as the physical size feature value of the squid target, and the physical size feature value of each squid target is mapped to a classification weight vector according to a preset size classification threshold. The mean value of the three-dimensional point cloud data of each squid target is calculated to obtain the three-dimensional center position coordinates of the squid target; Based on the visual classification confidence vector and the classification weight vector of each squid target, the classification probability vector of the squid target is calculated using the following formula: in, Indicates the first The classification probability vector of each squid target. The preset fusion weight coefficients, The first output of the model Visual classification confidence vectors for individual squid targets Indicates by the first The classification weight vector for each squid target; The category corresponding to the highest probability value in the classification probability vector is determined as the final classification label of the squid target. The three-dimensional center position, the final classification label and the instance segmentation mask of each squid target are aggregated to generate a target structured list.

5. The method according to claim 1, characterized in that, The process of assigning targets to be grasped to each idle robotic arm based on the target structured list, and planning the motion trajectory for each target to be grasped to generate a collision-free motion trajectory and an optimal grasping point includes: Iterate through the list of currently idle robotic arms and obtain the three-dimensional position of the end effector of each idle robotic arm in the list. The squid targets in the target structured list are identified as targets to be grabbed, and candidate grab points are obtained in the body region of the target to be grabbed based on the instance segmentation mask and the three-dimensional point cloud data of each target to be grabbed. Calculate the crawling quality score for each candidate crawling point, and determine the crawling point with the highest crawling quality score for each target to be crawled as the optimal crawling point; Based on the three-dimensional position of each end effector, calculate the Euclidean distance between the end position of each idle robotic arm and the optimal grasping point of all the targets to be grasped, and generate a target-robotic arm binding list based on the Euclidean distance and the preset binding requirements. Based on the target-robotic arm binding list, motion trajectory planning is performed on each bound target-robotic arm pair to generate a collision-free motion trajectory from the current position of the idle robotic arm to the optimal grasping point of the target to be grasped.

6. An AI-based robotic arm sorting control system, characterized in that, The system includes: The image acquisition module is used to acquire RGB images and depth images of squid targets in response to a target trigger signal within the recognition area of ​​the conveyor belt, and to perform normalization preprocessing on the RGB images and depth images of the squid targets to obtain preprocessed image data pairs. The data processing module is used to generate instance segmentation masks, visual classification confidence scores, and 3D point cloud data of squid targets based on the preprocessed image data input to a multi-task deep learning model. The list construction module is used to calculate the physical size features of each squid target based on the three-dimensional point cloud data, and generate a target structured list based on the physical size features, the visual classification confidence score and the instance segmentation mask; wherein, the target structured list includes the three-dimensional position, final classification label and instance segmentation mask of each squid target; The trajectory acquisition module is used to assign targets to be grasped to each idle robotic arm based on the target structured list, and to plan the motion trajectory for each target to be grasped, generating a collision-free motion trajectory and the optimal grasping point. The instruction generation module is used to generate sorting instructions based on the collision-free motion trajectory and the optimal gripping point; wherein, the gripping instruction is used to instruct the robotic arm assigned the task to sort the squid target according to the collision-free motion trajectory, the optimal gripping point and preset gripping parameters, and obtain sorting results.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images