A convolutional neural network vision sorting system and method for intelligent warehousing

Abstract

The application discloses a kind of convolutional neural network vision sorting systems and methods for intelligent warehousing, comprising the following steps: obtaining the order information to be sorted in the preset time period, according to the order of goods category and demand quantity, segmentation and reorganization is carried out, and a plurality of sorting sequence sets are generated;Configure the corresponding sorting equipment task;Obtain image features and point cloud features;Construct the goods recognition and pose estimation model based on convolutional neural network, combined with image features and point cloud features for joint training, output the category and spatial pose of goods;According to the recognition result and pose information, control the mechanical arm and end effector to complete the grabbing and sorting of goods.The application generates sorting sequence set by analyzing order information and performing segmentation and reorganization, allocates tasks in combination with demand quantity and priority, realizes the optimization scheduling of sorting process and efficient use of resources.Improve sorting accuracy and safety, reduce manual intervention and risk of wrong and missing sorting.

Description

Technical Field

[0001] This invention relates to the field of intelligent warehousing and robotic automation technology, specifically to a convolutional neural network visual sorting system and method for intelligent warehousing. Background Technology

[0002] With the rapid development of e-commerce, intelligent manufacturing, and new retail, warehousing and logistics systems are constantly evolving towards automation, intelligence, and efficiency. Traditional manual sorting methods can no longer meet the demands of current high-concurrency, multi-category, and fast-turnover warehousing operations, resulting in low efficiency, high error rates, and rising labor costs. Against this backdrop, intelligent warehousing systems are gradually becoming a core component of modern logistics systems, with automated sorting technology serving as a crucial link that directly impacts overall operational efficiency and service quality. In recent years, the deep integration of robotics and artificial intelligence has driven the rapid development of unmanned sorting systems, especially in e-commerce warehouses and express delivery distribution centers, where automated sorting systems based on industrial robots have shown broad application prospects.

[0003] In intelligent sorting systems, accurately and quickly identifying target goods and obtaining their spatial pose is a prerequisite for automated robotic grasping. Traditional sorting methods largely rely on barcode scanning or RFID tag identification. For example, the invention patent with publication number CN104933381B discloses an RFID-based intelligent mobile picking system and method. This system, which combines RFID tags and mobile picking carts in e-commerce warehouses, solves the problems of pickers not being able to predict the location of goods and high application costs. While these methods are low-cost, they require high integrity of the goods packaging and cannot handle situations such as missing tags, damaged tags, or stacked obstructions, resulting in poor adaptability. With the advancement of computer vision technology, visual perception-based sorting solutions have gradually become a research hotspot. In particular, the rise of deep learning technology has enabled convolutional neural networks (CNNs) to demonstrate superior performance in tasks such as image classification, object detection, and semantic segmentation, providing a new technical path for the automatic identification of goods in complex environments. By combining 3D information acquired by depth cameras, vision systems can achieve precise positioning of disordered stacked goods, significantly improving the flexibility and intelligence level of sorting systems.

[0004] To further improve sorting efficiency and system stability, modern intelligent warehousing systems typically adopt an architecture of multi-robot collaboration and parallel operation in multiple areas. However, without a reasonable task scheduling mechanism, problems such as equipment idling, task congestion, or resource contention can easily arise, ultimately reducing overall efficiency. Therefore, how to scientifically plan and dynamically allocate sorting tasks based on order characteristics has become a key factor affecting system performance. In existing systems, some use fixed priority or simple FIFO (First-In, First-Out) strategies for task dispatch, failing to fully consider the distribution of goods categories, the urgency of demand, and equipment load status, resulting in lengthy sorting paths and response delays. Furthermore, image processing and task scheduling modules often operate independently, lacking a collaborative optimization mechanism, making it difficult to achieve efficient end-to-end closed-loop control and limiting the overall system performance.

[0005] Despite the progress made in vision-based intelligent sorting technology, several technical bottlenecks remain. First, recognition methods relying solely on two-dimensional image information lack robustness under varying lighting conditions, reflective surfaces, or partial occlusion, easily leading to misidentification or pose estimation errors. While pure 3D point cloud processing methods can acquire geometric structure information, they underutilize texture and color features, limiting classification capabilities. Second, most systems fail to integrate order scheduling with visual perception optimization, and task allocation lacks anticipation of actual recognition difficulty and grasping feasibility, resulting in frequent idle runs or grasping failures. Finally, the closed-loop control from perception to execution lacks refined modeling and dynamic feedback mechanisms for the entire sorting process, making it difficult to achieve high-precision, high-stability continuous operation. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a convolutional neural network visual sorting system and method for intelligent warehousing, so as to overcome the defects of low recognition accuracy, unreasonable scheduling and poor system coordination in the prior art, and improve the overall performance of intelligent warehousing system.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A convolutional neural network-based visual sorting method for intelligent warehousing includes the following steps:

[0009] Obtain the order information to be sorted within a preset time period, and divide and reorganize the orders according to the product category and demand to generate multiple sorting sequence sets;

[0010] Based on the demand and sorting priority of goods in the sorting sequence, sorting levels are set and assigned to different sorting areas, and corresponding sorting equipment tasks are configured.

[0011] Collect image information of goods to be sorted, extract regions of interest after image preprocessing, and obtain image features and point cloud features;

[0012] A cargo recognition and pose estimation model based on convolutional neural networks is constructed. The model is jointly trained by combining image features and point cloud features to output the cargo category and spatial pose.

[0013] Based on the recognition results and pose information, the robotic arm and end effector are controlled to complete the grasping and sorting of goods.

[0014] A convolutional neural network-based visual sorting method for intelligent warehousing is proposed. First, the system acquires order information for sorting within a preset time period. Based on the product category and demand volume, the original orders undergo semantic parsing and data reconstruction. Similar product categories or frequently co-occurring product combinations are clustered and merged to generate multiple structured sorting sequence sets, thereby reducing repetitive handling and path conflicts. Subsequently, sorting priorities are set according to indicators such as the urgency of demand and outbound frequency of goods in each sequence set. Sorting levels are dynamically divided based on the equipment load of the sorting area, rationally allocating tasks to different physical areas to achieve optimized resource allocation and parallel operation. In the perception stage, color images and depth information of the goods to be sorted are acquired using a depth camera. After preprocessing such as denoising and alignment, regions of interest (ROIs) are extracted. Two-dimensional image texture features and three-dimensional point cloud geometric features are extracted separately to form complementary and fused multimodal inputs. Based on this, a joint recognition model based on a convolutional neural network is constructed. Through end-to-end collaborative training of image and point cloud features, high-precision output of product category and six-degree-of-freedom spatial pose is achieved. Ultimately, the system converts the recognition results into pose commands that the robotic arm can execute, driving the end effector to complete precise grasping and directional delivery, forming a complete closed-loop control from task planning to physical execution.

[0015] As a preferred method, the step of obtaining the order information to be sorted within a preset time period, and segmenting and reorganizing the orders according to the product category and demand to generate multiple sorting sequence sets specifically includes:

[0016] Extract product category keywords and quantity information from the order to generate a product data sequence;

[0017] Calculate the similarity between different product data sequences, and merge sequences with similarity higher than a preset threshold into the same sorting sequence set;

[0018] The sorting sequence is optimized by sorting the goods according to their average demand time and optimizing the sorting path and task scheduling order.

[0019] As a preferred method, image information of the goods to be sorted is acquired using a depth camera. After image preprocessing, the region of interest is extracted to obtain image features and point cloud features, specifically including:

[0020] Color and depth images were acquired using a structured light depth camera;

[0021] The image is denoised by median filtering and spatial nearest neighbor filtering, and the depth image and color image are aligned.

[0022] The corner projection regions in the image are extracted based on the convolutional neural network, and the three-dimensional corner reconstruction is performed by combining the point cloud information.

[0023] By using edge detection and point cloud segmentation algorithms, the dense center of the point cloud of the target object is extracted to obtain candidate grabbing regions.

[0024] As a preferred approach, the construction of a cargo recognition and pose estimation model based on a convolutional neural network, which combines image features and point cloud features for joint training, outputs the cargo category and spatial pose, specifically includes:

[0025] An improved YOLO-V2 network structure is adopted, with a 416×416×3 color image as input and the output being the two-dimensional projected coordinates of multiple corner points and their confidence scores.

[0026] Semi-automatic annotation was performed using Apriltags visual labeling tools to generate a high-precision training dataset.

[0027] The pose of the target object relative to the camera coordinate system is solved by combining the known 3D corner coordinates with the 2D projection points predicted by the network using the PnP algorithm.

[0028] Nonmaximum suppression and point cloud clustering algorithms are introduced to filter and correct the predicted pose, thereby improving the accuracy of pose estimation.

[0029] As a preferred method, when obtaining the final capture point coordinates, a weighted fusion method is used to correct the network prediction bias, the expression of which is:

[0030]

[0031] in, The final capture point coordinates; The 3D point coordinates obtained by backprojecting the predicted corner points of the network; The coordinates of the dense center of the point cloud obtained from point cloud segmentation; The fusion weighting coefficient has a range of values. .

[0032] As a preferred method, before performing the crawling, it is necessary to verify the consistency between the network predicted points and the point cloud center. The criterion is as follows:

[0033]

[0034] in, The distance between the two points is Euclidean distance; when the criterion is met, the pose estimation result is deemed reliable, and the grasping operation is allowed.

[0035] As a preferred method, before performing the grabbing operation, a surface geometric stability score of the candidate grabbing area is calculated to select the optimal grabbing point; the stability score is calculated using the following formula:

[0036]

[0037] in, The surface geometric stability score is given, with a value range of [value range missing]. A larger value indicates that the area is more suitable for stable crawling; This represents the number of point cloud points sampled within the candidate capture area; For the first Unit normal vector of each point cloud point; This is the average vector of the normal vectors of all point cloud points within the candidate region; It is the Euclidean norm of the vector difference, reflecting the degree of deviation of the local normal. For the first The curvature value of the neighborhood of a point in the point cloud; This is the curvature penalty coefficient, used to adjust the intensity of the curvature's influence on the score; its value ranges from [value missing]. ; This is the exponential decay factor for the effect of curvature;

[0038] when If the candidate grab area is determined to meet the geometric stability requirements, the grab operation is allowed.

[0039] As a preferred method, the step of controlling the robotic arm and end effector to complete the grasping and sorting of goods based on the recognition results and pose information specifically includes:

[0040] The target object's pose information is published to the robotic arm control node via the ROS system; joint control commands are generated through inverse kinematics calculation to drive the robotic arm to move to the gripping point; the pneumatic suction cup gripper is controlled to complete the adsorption and release of goods; and autonomous navigation and positioning between multiple shelves are achieved by combining with the omnidirectional mobile chassis.

[0041] A convolutional neural network-based visual sorting system for intelligent warehousing includes:

[0042] Depth camera module, used to acquire color and depth images of goods to be sorted;

[0043] The image processing module is used to extract image features and point cloud features, and to perform pose estimation;

[0044] The robotic arm control module is used to control the robotic arm to complete the grasping operation based on the pose information;

[0045] The mobile chassis module is used to enable the sorting platform to move autonomously between multiple shelves;

[0046] The ROS communication module is used for data interaction between various hardware devices and software nodes.

[0047] As a preferred embodiment, the image processing module further includes: a convolutional neural network unit for corner projection prediction and cargo recognition; a point cloud processing unit for depth image alignment, edge detection, and point cloud segmentation; a pose calculation unit for solving the pose of the target object using the PnP algorithm; and a grasping point correction unit for performing weighted fusion calculation.

[0048] The present invention has at least the following beneficial effects:

[0049] By acquiring order information to be sorted within a preset time period and segmenting and reorganizing orders according to product category and demand, multiple sorting sequence sets are generated, enabling refined organization and optimized scheduling of sorting tasks. Furthermore, by combining product demand and sorting priority to set sorting levels and allocate them to different sorting areas, sorting equipment tasks are rationally configured, significantly improving the overall operational efficiency and resource utilization of the sorting system. At the perception level, by collecting image information of goods to be sorted, preprocessing it, extracting regions of interest, and fusing image features and point cloud features, the system enhances the comprehensive representation capability of the appearance, texture, and three-dimensional geometric information of goods in complex warehousing environments, effectively improving the robustness of recognition under varying lighting conditions, partial occlusion, or stacking scenarios. A convolutional neural network-based goods recognition and pose estimation model is constructed. Through joint training of image and point cloud features, high-precision goods category discrimination and six-degree-of-freedom spatial pose estimation are achieved, providing reliable input for subsequent accurate grasping. Ultimately, based on the recognition results and pose information, the robotic arm and end effector are controlled to complete the grasping and sorting operations, realizing closed-loop automation from task planning to execution control, which greatly improves sorting accuracy and operational safety, and reduces the need for manual intervention and the risk of missorting or omission. Attached Figure Description

[0050] To reveal the technical details of the embodiments of the present invention, the accompanying drawings involved in the embodiments will be briefly described below. It should be emphasized that these drawings only present several embodiments of the present invention and should not be considered as defining the scope of the invention. For those skilled in the art, other related drawings can still be derived based on these drawings without inventive effort.

[0051] Figure 1 This is a schematic diagram of a convolutional neural network visual sorting method for intelligent warehousing.

[0052] Figure 2This is a schematic diagram of a convolutional neural network visual sorting system for intelligent warehousing.

[0053] Figure 3 This is a schematic diagram of the model training and inference process in the embodiment. Detailed Implementation

[0054] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0055] In the following sections, embodiments of the present disclosure will be described in detail with the aid of the accompanying drawings. However, it should be understood that the present disclosure is not limited to the specific forms shown herein. Rather, it should be understood to encompass various variations, equivalents, and / or alternatives to the embodiments of the present disclosure. In illustrating the drawings, the same reference numerals will be used to denote similar components.

[0056] In this context, the use of the singular form also covers the plural form, unless otherwise explicitly stated in the text. In this explanation, it should be understood that terms such as "comprising" or "possessing" are intended to indicate the presence of a feature, quantity, step, operation, structural component, part, or combination thereof, without precluding the possibility or addition of one or more other features, quantities, steps, operations, structural components, parts, or combinations thereof.

[0057] It should be clarified that while the following description provides detailed specific information to aid in a comprehensive understanding of the exemplary embodiments, those skilled in the art will recognize that the exemplary embodiments can be implemented even without these specific details. For example, the system may be illustrated using block diagrams to avoid excessive detail that could obscure the clarity of the example. In other cases, to maintain the clarity of the example, unnecessary details of well-known processes, structures, and techniques may be omitted.

[0058] like Figure 1 As shown, a convolutional neural network-based visual sorting method for intelligent warehousing includes the following steps:

[0059] S1: Obtain the order information to be sorted within a preset time period, and divide and reorganize the orders according to the product category and demand to generate multiple sorting sequence sets;

[0060] S2: Based on the demand and sorting priority of goods in the sorting sequence, set sorting levels and allocate them to different sorting areas, and configure the corresponding sorting equipment tasks;

[0061] S3: Collect image information of goods to be sorted, extract regions of interest after image preprocessing, and obtain image features and point cloud features;

[0062] S4: Construct a cargo recognition and pose estimation model based on convolutional neural networks, combine image features and point cloud features for joint training, and output the cargo category and spatial pose.

[0063] S5: Based on the recognition results and pose information, control the robotic arm and end effector to complete the grasping and sorting of goods.

[0064] This invention provides a convolutional neural network-based visual sorting method for intelligent warehousing, organically combining order scheduling, visual recognition, and robotic execution to achieve efficient, accurate, and automated goods sorting. The process begins with order information. The system collects all pending orders for a future period, analyzes the types and quantities of goods required, and uses clustering and recombination algorithms to group orders with similar characteristics into multiple sorting sequence sets. This is equivalent to pre-planning which goods should be picked together and in what order is most efficient, thereby reducing repetitive tasks and path conflicts. Based on this, the system sets different sorting priorities according to the urgency, outbound frequency, and overall demand of goods in each sequence set, and assigns them to appropriate sorting areas. Simultaneously, it matches corresponding robotic arms, conveyor belts, and other equipment resources to ensure that tasks are matched with capabilities, improving overall operational efficiency.

[0065] Next, the system moves into the perception phase. It uses a depth camera to capture images of the stacked goods, acquiring multimodal data containing color and depth information. These raw images undergo preprocessing operations such as filtering and alignment to remove noise interference and extract the region of interest containing the target goods. Subsequently, the system extracts visual features such as texture and contour from the color images, while simultaneously extracting 3D geometric structure information from the point cloud data. These two types of features complement each other, enabling the system to not only "see" the appearance of the goods but also "perceive" their three-dimensional shape and spatial position, greatly enhancing its recognition capabilities in complex scenes (such as partial occlusion, disordered stacking, and uneven lighting).

[0066] To achieve high-precision identification and positioning, the system employs a convolutional neural network to construct a fusion model. This model, trained on a large amount of labeled data, learns both image and point cloud features simultaneously, ultimately outputting the category of each item and its precise spatial location and orientation. This joint training method is more robust than using images or point clouds alone, and is particularly suitable for unordered, mixed-placement warehouse environments. Finally, the identification results are transmitted to the control system. The robotic arm automatically plans its motion path based on the spatial pose of the goods, adjusts the attitude and force of the end effector, completes the grasping action, and accurately places the goods onto the designated container or conveyor belt, achieving fully automated operation from "seeing" to "grabbing." The entire method is logically clear and progressively advanced, not only improving sorting speed and accuracy but also significantly reducing reliance on manual labor, making it suitable for the large-scale, highly dynamic operational needs of modern smart warehousing.

[0067] In a preferred embodiment, the step of obtaining the order information to be sorted within a preset time period, and segmenting and reorganizing the orders according to the product category and demand to generate multiple sorting sequence sets specifically includes:

[0068] Extract product category keywords and quantity information from the order to generate a product data sequence;

[0069] Calculate the similarity between different product data sequences, and merge sequences with similarity higher than a preset threshold into the same sorting sequence set;

[0070] The sorting sequence is optimized by sorting the goods according to their average demand time and optimizing the sorting path and task scheduling order.

[0071] In intelligent warehousing and sorting processes, dealing with a large number of scattered and mixed orders directly leads to repetitive walking, idle equipment, and low efficiency. To address this, the system first acquires information on all orders to be sorted within a future timeframe. Each order is then structured and parsed, extracting the product category names and corresponding quantities to form a sequence of product data. This essentially "translates" each order into a machine-understandable goods list. Subsequently, the system calculates the similarity between different orders. For example, if two orders contain a large number of the same items—order A requires 10 cardboard boxes and 5 plastic bottles, while order B requires 8 cardboard boxes and 6 plastic bottles—their categories and quantities are similar, and they are considered highly similar. When the similarity exceeds a set threshold, the system groups these orders into the same sorting sequence set, essentially packaging "incidental" tasks together for batch picking, reducing the number of back-and-forth handling operations. Finally, the system analyzes the average outbound time or urgency of goods in each sorting sequence, prioritizing tasks that are about to expire or have high frequency of outbound processing. It sorts the task queue according to time urgency, thereby optimizing the robotic arm's work sequence and movement path, avoiding uneven workloads or task backlogs. Through this series of operations, the system achieves a shift from "passively responding to orders" to "proactively planning tasks," significantly improving overall sorting efficiency and resource utilization.

[0072] In a preferred embodiment, image information of the goods to be sorted is acquired using a depth camera. After image preprocessing, the region of interest is extracted to obtain image features and point cloud features, specifically including:

[0073] Color and depth images were acquired using a structured light depth camera;

[0074] The image is denoised by median filtering and spatial nearest neighbor filtering, and the depth image and color image are aligned.

[0075] The corner projection regions in the image are extracted based on the convolutional neural network, and the three-dimensional corner reconstruction is performed by combining the point cloud information.

[0076] By using edge detection and point cloud segmentation algorithms, the dense center of the point cloud of the target object is extracted to obtain candidate grabbing regions.

[0077] To accurately identify haphazardly stacked goods in a warehouse and locate suitable grabbing positions, the system first uses a structured light depth camera to simultaneously acquire color and depth images of the goods. This type of camera not only captures color and texture information but also records the distance of each pixel to the camera, forming 3D point cloud data, providing a rich visual foundation for subsequent analysis. The acquired raw images often contain noise or distortion, such as blurring or depth jumps caused by uneven lighting. Therefore, the system employs median filtering and spatial nearest neighbor filtering to denoise and improve image quality. Simultaneously, it precisely aligns the color and depth images spatially, ensuring a one-to-one correspondence between color information and 3D position, avoiding misalignment. Based on this, the system automatically identifies key feature regions of the goods in the images using a trained convolutional neural network, such as corner points or edge contours of containers. These corner points have distinct features in the 2D image; combined with the corresponding depth information, their true positions in 3D space can be deduced, achieving 3D reconstruction of the corner points and providing crucial reference points for subsequent precise positioning. Next, the system further utilizes edge detection and point cloud segmentation algorithms to separate the point cloud region of a single item from complex backgrounds and stacked environments, and calculates the center position where the point cloud distribution is densest. This position is usually located near the object's center of gravity and is the most stable and safest candidate gripping point for the robotic arm. The entire process, from original image acquisition to candidate gripping region extraction, is progressive, utilizing both the image's detailed texture and incorporating the three-dimensional spatial structure. This allows the system to reliably find the target area to grip even in complex scenarios such as occlusion, reflection, or stacking, laying a solid foundation for stable robot operation.

[0078] In a preferred embodiment, the construction of a cargo recognition and pose estimation model based on a convolutional neural network, which combines image features and point cloud features for joint training, and outputs the cargo category and spatial pose, specifically includes:

[0079] An improved YOLO-V2 network structure is adopted, with a 416×416×3 color image as input and the output being the two-dimensional projected coordinates of multiple corner points and their confidence scores.

[0080] Semi-automatic annotation was performed using Apriltags visual labeling tools to generate a high-precision training dataset.

[0081] The pose of the target object relative to the camera coordinate system is solved by combining the known 3D corner coordinates with the 2D projection points predicted by the network using the PnP algorithm.

[0082] Nonmaximum suppression and point cloud clustering algorithms are introduced to filter and correct the predicted pose, thereby improving the accuracy of pose estimation.

[0083] To achieve high-precision identification and positioning of warehouse goods, a recognition and pose estimation model based on convolutional neural networks was constructed. This model combines two-dimensional image information with three-dimensional spatial structure to improve the accuracy of recognition and the stability of positioning. The system first employs an improved YOLO-V2 network structure, using a 416×416 color image as input. Through deep learning, it automatically extracts the visual features of the goods and quickly predicts the two-dimensional projection positions of the goods at multiple key corner points in the image and their reliability. This approach can not only identify the category of goods but also locate their specific outline in the image. (See also...) Figure 3 To improve the accuracy and efficiency of training data, the system utilizes Apriltags visual labeling for semi-automatic annotation: Apriltags calibration boards of known size and location are placed near the goods, using their precise corner information as a reference to assist in annotating the corner positions of the actual goods. This generates a high-quality, highly consistent training sample set, significantly reducing the workload and error of manual annotation. During actual operation, these two-dimensional corner positions output by the network correspond one-to-one with the pre-known three-dimensional corner coordinates of the goods. The PnP (Perspective n-Point) algorithm is used to calculate the precise position and pose of the goods relative to the camera, i.e., how much it has shifted and rotated in space. Finally, since occlusion or false detections may exist in the real environment, the initially estimated pose may not be stable enough. Therefore, the system introduces non-maximum suppression and point cloud clustering algorithms to filter and fuse multiple candidate poses, eliminating abnormal results and retaining the most reasonable set of pose parameters, thereby further improving the positioning accuracy. The entire process, from image recognition to spatial positioning and then to result optimization, is progressive and achieves rapid, accurate, and reliable identification and pose estimation of goods in complex warehousing environments, providing key support for the subsequent precise grasping by the robotic arm.

[0084] In one embodiment, it includes:

[0085] Image acquisition and preprocessing:

[0086] In a smart warehousing environment, to achieve cargo recognition and pose estimation, the stacked goods first need to be photographed using robots or fixed vision systems equipped with RGB cameras. Each captured image is uniformly scaled and cropped to a 416×416×3 color image (RGB three channels), which is the standard input size for neural networks, ensuring the consistency of input data and the effectiveness of model training. To improve the stability and accuracy of subsequent detection processes, these images also need to undergo a series of preprocessing operations, including noise reduction, white balance correction, and contrast enhancement, to eliminate factors that may affect the accuracy of keypoint detection.

[0087] Network architecture design: Improved YOLO-V2 for key point detection

[0088] Traditional YOLO-V2 networks are primarily used for object detection tasks, outputting bounding boxes and class labels. However, we structurally improve upon this by replacing the output head with a multi-keypoint regression head. This allows the network to no longer predict rectangular bounding boxes, but instead directly regress the 2D projection coordinates of the eight 3D corner points of a regular item (such as a standard cardboard box) onto the image plane. Specifically, the network output is a feature map where each spatial location corresponds to a region in the original image, and it predicts whether that region contains a certain corner point and its coordinates. and confidence level Ultimately, the network will output a total of 8 sets. , representing the image positions and detection confidence of the eight vertices of the cardboard box, respectively, thus providing accurate position information for subsequent pose estimation.

[0089] Training data generation: Apriltags visual labeling aids semi-automatic annotation

[0090] Considering the time-consuming and inaccurate nature of manual labeling of cargo corners, we adopted a method combining Apriltags visual labeling and geometric constraints to achieve semi-automatic high-precision labeling. The specific process is as follows: First, the calibration environment is set up by placing an Apriltags labeling board of known size near the carton to be identified. Then, images of the carton and labeling board are captured from different angles. Next, Apriltags is used to solve the camera pose relative to the labeling board, and based on this, the two-dimensional projected coordinates of the carton corners are calculated. Finally, a labeled dataset containing "images + 8 corner coordinates + visibility labels" is constructed to improve the training of the YOLO-V2 network.

[0091] Model training and inference

[0092] After data labeling, the improved YOLO-V2 network was trained end-to-end using the aforementioned high-quality training dataset. During training, the loss function consisted of coordinate regression loss (e.g., SmoothL1 Loss), confidence classification loss (e.g., BCELoss), and visibility judgment loss to ensure the model learned accurate keypoint locations, reliable confidence scores, and information on corner visibility. In practical applications, when a new image is input to the trained model, the network can directly output the predicted locations and confidence scores of the eight corner points of the cargo, without relying on the Apriltags label palette, thus achieving automation and intelligence in cargo recognition and pose estimation.

[0093] In one specific embodiment, the system needs to automatically identify and estimate the spatial pose of a standard rectangular cardboard box to guide the robotic arm in precise grasping. The box measures 400 mm in length, 300 mm in width, and 200 mm in height. Its three-dimensional geometric model, including the three-dimensional coordinates of eight corner points in the object's own coordinate system, has been pre-entered into the system database. For example, with the center of the bottom surface as the origin, the coordinates of the four bottom corner points are: A(-200,-150,0), B(200,-150,0), C(200,150,0), and D(-200,150,0), in millimeters. These three-dimensional points constitute the geometric foundation of the object and are crucial inputs for subsequent pose calculations.

[0094] When a cardboard box is placed in the sorting area, the system acquires its color image using a structured light depth camera with a resolution of 640×480 pixels. An improved YOLO-V2 network processes the image, detecting the key corner points of the cardboard box and outputting their two-dimensional projection positions in the image. For example, the network predicts the pixel coordinates of the four bottom corner points as: a(320,240), b(410,250), c(400,330), and d(310,320). These two-dimensional points are the visual projections of the three-dimensional corner points from the current viewpoint, reflecting the actual appearance of the object in the image. Meanwhile, the camera was calibrated before deployment, and its intrinsic parameter matrix (including focal length fx=500, fy=500, optical center cx=320, cy=240, where fx represents the equivalent focal length of the camera in the horizontal axis (u-axis) of the image; fy represents the equivalent focal length of the camera in the vertical axis (v-axis) of the image; cx represents the coordinates of the principal point in the horizontal axis; and cy represents the coordinates of the principal point in the vertical axis) is known, ensuring the accuracy of subsequent geometric calculations.

[0095] To recover the precise position and orientation of the cardboard box in space from these 2D-3D corresponding points, the system employs the Perspective-n-Point (PnP) algorithm framework for pose determination. The PnP problem refers to solving for the rotation matrix and translation vector of an object relative to the camera coordinate system, i.e., six-degree-of-freedom pose (6DoF), given n 3D spatial points and their 2D projection points in the image, and known camera intrinsic parameters. In this embodiment, the EfficientPnP (EPnP) algorithm is used as the specific implementation method for the PnP problem to improve computational efficiency and stability.

[0096] The EPnP algorithm represents n 3D points on an object as a weighted combination of four virtual control points, thus transforming the original pose estimation problem into solving for these four control points. Specifically, let any point on the object... The three-dimensional coordinates in the object coordinate system are: (i=1,2,…,n), then it can be expressed as:

[0097]

[0098] in, j=1,2,3,4) represents the three-dimensional coordinates of the four virtual control points in the object coordinate system; Let the homogeneous barycenter coordinates of the i-th corner point relative to the j-th control point satisfy the following condition: The weighting coefficients are pre-calculated and stored in the model offline using singular value decomposition (SVD), and do not change with the object's pose.

[0099] In actual operation, the system first reads the weight coefficients corresponding to each corner point based on the known set of 3D model points. Subsequently, in the camera coordinate system, let the spatial coordinates of these four control points be... (j=1,2,3,4), then the estimated position of the i-th 3D point in the camera coordinate system is:

[0100]

[0101] in, This represents the three-dimensional spatial coordinates of the i-th corner point in the camera coordinate system; by utilizing the linear combination relationship of the control points, the overall pose estimation of the object is transformed into solving for the four control points.

[0102] Next, Through camera intrinsic parameter matrix Perform perspective projection to obtain its predicted position on the image plane:

[0103]

[0104] in, This represents the predicted projected pixel coordinates of the i-th corner point; The perspective projection function is represented by the following homogeneous coordinate projection equation:

[0105]

[0106] in For depth scaling factor, The camera intrinsic parameter matrix is ​​known.

[0107] The system iteratively adjusts through nonlinear optimization algorithms (such as Levenberg-Marquardt). The value of is to minimize the reprojection error of all corner points, that is:

[0108]

[0109] in, This represents the actual 2D pixel coordinates of the i-th corner point detected by the convolutional neural network; the optimization objective is to make the predicted projection point as close as possible to the detection result.

[0110] When the four control points are in the optimal position in the camera coordinate system Once determined, the system can construct the rigid body transformation from the object coordinate system to the camera coordinate system. The optimal rotation matrix is ​​then solved using singular value decomposition (SVD). (3×3 orthogonal matrix) and translation vector This ultimately yields the six-DOF pose of the cardboard box relative to the camera. If the calculated result is a translation vector... This indicates that the center of the carton is located 800 mm in front of the camera, offset 100 mm to the left, and 300 mm downwards; the Euler angles corresponding to the rotation matrix are 15° around the Z-axis and 5° around the Y-axis, indicating that the carton is tilted to the right and forward.

[0111] The pose information is sent to the robotic arm control system, which, combined with inverse kinematics calculations, plans a collision-free trajectory and adjusts the end effector's posture to be parallel to the carton surface, ultimately achieving stable grasping. In summary, this embodiment employs a technical approach that utilizes deep learning to extract 2D corner points, known 3D model points, the PnP framework, and efficient EPnP solution, fully leveraging the advantages of the EPnP algorithm in computational efficiency, numerical stability, and noise resistance. Compared to traditional PnP methods, EPnP eliminates the need for initial pose guessing, has low computational complexity (O(n)), and fast convergence speed, making it particularly suitable for high-frequency, dynamic sorting tasks in intelligent warehousing, achieving high-precision vision-guided grasping.

[0112] In a preferred embodiment, when obtaining the final capture point coordinates, a weighted fusion method is used to correct the network prediction bias, the expression of which is:

[0113]

[0114] in, The final capture point coordinates; The 3D point coordinates obtained by backprojecting the predicted corner points of the network; The coordinates of the dense center of the point cloud obtained from point cloud segmentation; The fusion weighting coefficient has a range of values. .

[0115] In acquiring the final grasping point coordinates, the system employs a weighted fusion method to correct the network prediction results, thereby improving positioning accuracy. The deep learning network can quickly identify key corner points of the cargo based on the image and, combined with camera parameters, back-project them into 3D space to obtain a preliminary 3D position estimate. However, due to factors such as lighting, occlusion, or model similarity affecting the image, the network prediction may have some bias. To compensate for this deficiency, the system simultaneously utilizes point cloud data collected by a depth camera. By segmenting the point cloud of the target area, it extracts the point set belonging to the cargo and calculates its spatial density center. This center position reflects the actual geometric center of the object in 3D space, possessing high measurement reliability. Based on this, the system fuses the 3D points predicted by the network with the point cloud density center, assigning different weights to both. Since the point cloud result is more stable, it is assigned a higher weight; while the network result retains a certain contribution to leverage its semantic recognition advantages. The final generated grasping point coordinates balance recognition accuracy and enhance resistance to noise and errors, providing a reliable basis for the robotic arm's precise grasping.

[0116] In a preferred embodiment, before performing the crawling, it is necessary to verify the consistency between the network predicted points and the point cloud center. The criterion is as follows:

[0117]

[0118] in, The distance between the two points is Euclidean distance in millimeters (mm). When the criterion is met, the pose estimation result is deemed reliable, and the grasping operation is allowed.

[0119] Before performing the grasping operation, the system verifies the reliability of the pose estimation results to ensure the accuracy and safety of the grasping process. Since the prediction results of deep learning networks can be affected by image quality, occlusion, or lighting variations, while the geometric centers obtained from point cloud segmentation typically have high spatial accuracy, the system judges the reliability of the results by comparing the consistency of the 3D positions from these two sources. Specifically, the 3D coordinates obtained by back-projecting the keypoints predicted by the network are compared with the dense centers calculated after point cloud segmentation, and the actual distance between them is calculated. If this distance is small, it indicates a high degree of agreement between the two different modalities, meaning that the network's recognition is basically consistent with the actual object position. In this case, the current pose estimation result is deemed reliable, and the robotic arm is allowed to continue the grasping action. Conversely, if the deviation is too large, there may be a recognition error or data anomaly, and the system will refuse to grasp and trigger a re-examination mechanism. This cross-validation method effectively improves the system's judgment accuracy and avoids grasping failures caused by errors from a single data source.

[0120] In a preferred embodiment, before performing the grasping operation, a surface geometric stability score of the candidate grasping region is calculated to screen for the optimal grasping point; the stability score is calculated using the following formula:

[0121]

[0122] in, The surface geometric stability score is given, with a value range of [value range missing]. A larger value indicates that the area is more suitable for stable crawling; This represents the number of point cloud points sampled within the candidate capture area, expressed in units of points. For the first Unit normal vector of each point cloud point; It is the average vector of the normal vectors of all point cloud points within the candidate region, and the normalized unit vector. It is the Euclidean norm of the vector difference, reflecting the degree of deviation of the local normal. For the first The curvature value of the neighborhood of each point cloud point, in millimeters (mm⁻¹). This is the curvature penalty coefficient, measured in millimeters (mm), used to adjust the intensity of the curvature's influence on the score. Its value range is... ; The exponential decay factor of the curvature effect; when If the candidate grab area is determined to meet the geometric stability requirements, the grab operation is allowed.

[0123] Before performing the grasping operation, the system not only verifies the consistency between the network's predicted points and the point cloud center, but also calculates the surface geometric stability score of the candidate grasping regions to select the optimal grasping point. This score comprehensively considers the surface flatness and curvature variation of the grasping region to ensure that the robotic arm can stably grasp the target object. Specifically, the system first samples the point cloud data within the candidate grasping regions and calculates the normal vector of each point and the curvature value of its local neighborhood. Then, by comparing the normal vector of each point with the average normal vector of all points in the region, the flatness of the region is evaluated: if the normal vectors of most points are close to the average normal vector, it indicates that the region is relatively flat and suitable for grasping. At the same time, the system also considers the influence of curvature, that is, the degree of curvature of the neighborhood where the point is located; the greater the curvature, the more uneven the surface, and the higher the risk during grasping. To quantify these factors, the system introduces an exponential decay factor to adjust the intensity of the curvature's influence on the score, so that the score of high curvature regions is significantly reduced. Finally, the system calculates a comprehensive surface geometric stability score based on the flatness and curvature information of all points. This score is dimensionless and ranges from 0 to 1; a higher value indicates that the area is more suitable for stable grasping. When the score exceeds 0.7, the system determines that the candidate grasping area meets the geometric stability requirements and allows the robotic arm to perform the grasping operation. This multi-level evaluation mechanism effectively improves the reliability and success rate of the grasping process, avoiding grasping failures caused by surface unevenness or excessive curvature.

[0124] In a preferred embodiment, the step of controlling the robotic arm and end effector to complete the grasping and sorting of goods based on the recognition result and pose information specifically includes:

[0125] The target object's pose information is published to the robotic arm control node via the ROS system; joint control commands are generated through inverse kinematics calculation to drive the robotic arm to move to the gripping point; the pneumatic suction cup gripper is controlled to complete the adsorption and release of goods; and autonomous navigation and positioning between multiple shelves are achieved by combining with the omnidirectional mobile chassis.

[0126] After completing the identification and pose estimation of the goods, the system uses ROS (Robot Operating System) to uniformly schedule the obtained target position and posture information, enabling collaborative work between modules. First, the pose information is encapsulated into a standard message format and published to the control node of the robotic arm as input instructions for the grasping task. Upon receiving this information, the robotic arm controller, combined with the preset grasping posture, uses inverse kinematics calculation to convert the target position and orientation of the end effector into the motion angles of each joint, thereby generating corresponding joint control instructions to drive the robotic arm to move smoothly and accurately to the predetermined grasping point. The role of inverse kinematics is to calculate the required rotation angle of each joint based on the length of each arm segment, joint type, and range of motion (these parameters are modeled and stored in the control system at the factory) to ensure the end effector accurately reaches the desired position and maintains the specified orientation.

[0127] Taking a typical three-axis articulated robotic arm as an example, suppose the system identifies the gripping point of the cardboard box as being located 800 mm in front of the camera, 100 mm to the right, and 600 mm above it, requiring the suction cup to grip vertically downwards. Upon receiving this target pose, the robotic arm controller first determines the required spatial position and direction of the end effector by combining it with the pre-set gripping posture for the task (e.g., the end effector pointing vertically downwards). Next, through inverse kinematics calculations, the end effector target is decomposed into the angles that each joint should rotate: for example, the first joint (base) needs to rotate 15 degrees to align with the target to the right, the second joint (upper arm) needs to rise 30 degrees, and the third joint (lower arm) needs to bend 45 degrees to accurately reach the designated position. These angle commands are sent to the servo motors of each joint, driving the robotic arm to move smoothly and coordinately to the gripping point. The entire process requires no manual teaching, achieving automatic and precise positioning.

[0128] Once the robotic arm reaches the target location, the system activates the pneumatic suction cup gripper at its end. By controlling the solenoid valve to open and close the air path, negative pressure suction is achieved to firmly grasp the surface of the goods. After grasping, the negative pressure is released at the target shelf or conveyor line location to place the goods. Throughout the process, the robotic arm is mounted on an omnidirectional mobile chassis with flexible steering and precise positioning capabilities. This chassis can automatically plan paths according to the sorting task, autonomously navigate and precisely position itself between multiple shelves, thus achieving fully automated operation from identification and grasping to handling and placement. This integrated control method effectively improves the response speed and operating efficiency of the sorting system and is suitable for complex and dynamic warehouse environments.

[0129] In a preferred embodiment, when the pneumatic suction cup gripper completes the adsorption operation of the goods, the vacuum pressure needs to be adjusted according to the surface material of the goods. For highly reflective or fragile packaged goods, the adsorption force should be reduced to avoid damage.

[0130] When controlling the pneumatic suction cup gripper to pick up goods, the system dynamically adjusts the vacuum pressure according to the different surface materials of the goods to ensure reliable and safe gripping. Since different packaging materials have varying tolerances to suction force, applying the same vacuum intensity to all goods may cause fragile surfaces such as highly reflective plastics, thin cardboard boxes, or easily damaged packaging to deform or even break. Therefore, while identifying the type of goods, the system calls upon a preset material parameter library to determine its surface characteristics, such as whether it is smooth, fragile, or coated. For sturdy packaging such as ordinary corrugated cardboard boxes, standard suction force is maintained to ensure stable gripping; while for highly reflective or identified as fragile packaging, the operating pressure of the vacuum pump is actively reduced to decrease the negative pressure intensity between the suction cup and the surface, thereby avoiding damage caused by localized stress concentration. This adjustment process is precisely controlled by a solenoid valve to control the airflow and throttling, resulting in rapid response and high repeatability. This enables adaptive gripping of different goods, balancing operational efficiency and cargo integrity.

[0131] In a preferred embodiment, the method further includes real-time monitoring of the operational status and sorting volume of each sorting area, and dynamic adjustment of sorting tasks and equipment allocation, specifically including:

[0132] The location of the sorting platform and the stacking status of goods are acquired in real time by LiDAR and industrial control computer; the sorting sequence set is dynamically adjusted based on sorting efficiency and equipment load; when the recognition accuracy decreases, the data annotation and incremental model training process is automatically triggered to improve the model's adaptability to environmental changes.

[0133] This embodiment also features real-time monitoring and dynamic scheduling capabilities for the operational status of each sorting area, improving overall operational efficiency and stability. By continuously scanning the sorting platform with LiDAR deployed on-site, combined with point cloud data processing by an industrial control computer, the system can accurately acquire the location information of each workstation and the stacking status of goods, such as whether an area is full, congested, or idle. Based on this status information, the system comprehensively evaluates the current workload and sorting efficiency of each device, dynamically adjusts the execution order of pending tasks, and prioritizes assigning tasks to idle or high-capacity sorting units, avoiding local overload or idle periods, and achieving rational resource allocation. Simultaneously, if a decrease in visual recognition accuracy is detected during operation (e.g., due to changes in lighting, the appearance of new packaging types), the system automatically triggers an anomaly warning and initiates a data labeling and incremental model training process: recently collected problematic samples are labeled and added to the training set, and the recognition model is updated on a small scale to quickly adapt to environmental changes. The entire process requires no manual intervention, forming a closed-loop control of perception, decision-making, optimization, and learning, significantly improving the intelligence level and long-term operational capability of the sorting system.

[0134] In a preferred embodiment, the method further includes real-time monitoring of the operational status and sorting volume of each sorting area, and dynamic adjustment of sorting tasks and equipment allocation. Specifically, this includes constructing a sorting task allocation cost function to evaluate the overall cost of allocating tasks to different sorting areas, and selecting the area with the lowest cost to execute the task. The expression for the cost function is:

[0135]

[0136] in, Assign the task to the first The comprehensive cost of each sorting area, with a value range of [value missing]. ;

[0137] Indicates the first The current equipment load value of each sorting area, in units of tasks / min, is calculated by the industrial control computer as the processing rate of the assigned tasks per unit time. This indicates the maximum allowable load threshold set by the system, in units of tasks per minute (tasks / min). Exceeding this value is considered overload. Indicates the distance from the current position of the moving chassis to the [number]th position. The length of the navigation path for each sorting area is in meters (m). This represents the maximum possible distance between all sorting areas in the system, expressed in meters (m). Indicates allocation to the first After the region is defined, the estimated start time for the task is in seconds (s); This indicates the latest allowed start time for the task (calculated based on the order outbound time limit), in seconds (s). Represents the weighting coefficients, satisfying

[0138] ,and It is used to adjust the relative importance of load, distance and urgency, and can be configured according to the operation strategy.

[0139] When the system needs to select a sorting area for a new task, it calculates the cost of allocating it to each available area in turn. Select The corresponding region serves as the target execution unit. This model achieves multi-objective collaborative optimization of load balancing, minimum energy consumption, and timely response, avoiding the resource waste problem of "busy areas getting busier and idle areas being idle" in traditional scheduling.

[0140] In a preferred embodiment, after performing the grasping operation, if a grasping failure is detected (e.g., the force sensor feedback is less than a threshold or visual confirmation that the grasp was not taken), the system initiates an adaptive retry mechanism. The core of this mechanism is to calculate the predicted grasping success rate of the current candidate grasping point and adjust the strategy accordingly. The expression for the grasping success rate prediction model is:

[0141]

[0142] in, This represents the predicted success rate of crawling, with a value range of [0, 1]. This represents the surface geometric stability score of the candidate grasping region, with a value range of [0, 1], derived from the aforementioned formula. , The coefficient of kinetic friction of the surface material of goods is obtained through database queries or online learning; for example, the coefficient of kinetic friction for corrugated cardboard boxes is approximately [value missing]. Plastic film packaging is approximately ; This indicates the effective contact area between the pneumatic suction cup and the surface of the goods, expressed in square centimeters (cm²). 2 The area is obtained by calculating the projected area after segmenting the depth image. This represents the area gain coefficient, in units of... This is used to adjust the nonlinear enhancement effect of contact area on success rate, with a value range of [value missing]. It can be calibrated through experiments.

[0143] according to Different strategies are adopted for different values:

[0144] like If the success rate is deemed high, a maximum of two parameter fine-tuning retries are allowed (such as increasing the vacuum pressure by 10% or slightly shifting the gripping point).

[0145] like The system prompts for manual review or switching to an alternative gripping strategy (such as using a gripper).

[0146] like If the capture point is not found, the visual recognition process is restarted to find a new candidate region.

[0147] This mechanism enables a shift from blind retries to intelligent decision-making based on model predictions, significantly improving the system's autonomous operation capabilities in scenarios such as complex packaging and low-friction surfaces.

[0148] In a preferred embodiment, the sorting equipment includes a UR5 six-degree-of-freedom robotic arm, a pneumatic suction cup gripper, an omnidirectional moving chassis, and a Realsense D435 depth camera. All equipment is integrated and communicates through the ROS system to achieve multi-device collaborative operation.

[0149] The sorting equipment in this embodiment consists of a UR5 six-DOF robotic arm, a pneumatic suction cup gripper, an omnidirectional mobile chassis, and a Realsense D435 depth camera. All components are integrated and communicate in real-time through ROS (Robot Operating System) to collaboratively complete the tasks of identifying, moving, gripping, and sorting goods. The depth camera is responsible for collecting color images and depth information from the environment and sending the data to the industrial control computer for identifying the position and 3D pose of the goods. The robotic arm, based on the calculated target pose and inverse kinematics calculations, plans its trajectory and moves to the gripping point. The pneumatic suction cup gripper at the end automatically adjusts its suction force according to the material of the goods to ensure stable gripping without damaging the packaging. Simultaneously, the robotic arm, mounted on the omnidirectional mobile chassis, can move flexibly forward, backward, or laterally in confined spaces, achieving autonomous navigation and precise positioning between multiple shelves. All devices interact with each other through the ROS system using a publish / subscribe mechanism. For example, the camera publishes the target position, the robotic arm subscribes to pose commands, and the chassis synchronizes its own state, ensuring coordinated and efficient operation of all modules. This ROS-based integrated architecture not only improves the modularity and scalability of the system, but also achieves seamless collaboration between perception, decision-making, motion and execution, significantly enhancing the automation and intelligence of sorting operations.

[0150] In a preferred embodiment, a convolutional neural network visual sorting system for smart warehousing includes: a memory, a processor, and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements a convolutional neural network visual sorting method for smart warehousing as described above.

[0151] A convolutional neural network visual sorting system for smart warehousing (see...) Figure 2 ),include:

[0152] Depth camera module, used to acquire color and depth images of goods to be sorted;

[0153] The image processing module is used to extract image features and point cloud features, and to perform pose estimation;

[0154] The robotic arm control module is used to control the robotic arm to complete the grasping operation based on the pose information;

[0155] The mobile chassis module is used to enable the sorting platform to move autonomously between multiple shelves;

[0156] The ROS communication module is used for data interaction between various hardware devices and software nodes.

[0157] This vision-based sorting system is designed for smart warehousing scenarios, achieving automatic identification and precise grasping of goods through multi-module collaboration. The system first uses a depth camera module to acquire color and depth images of the goods to be sorted, obtaining appearance information and 3D spatial data of the target objects. The image processing module receives this data and uses a convolutional neural network to extract key features from the images, identifying the category and corner positions of the goods. It then combines this with point cloud data for 3D reconstruction and pose estimation, determining the precise position and orientation in space. Based on this pose information and a pre-set grasping strategy, the robotic arm control module generates joint motion commands through inverse kinematics calculations, driving the robotic arm to the target position and controlling the end effector suction cup to complete the adsorption and transport. Simultaneously, the mobile chassis module autonomously navigates to different shelf locations according to the sorting task requirements, expanding the robotic arm's working range and enabling continuous multi-area operation. All modules are uniformly scheduled through a ROS communication module, using standardized message formats to achieve real-time data interaction between the camera, robotic arm, chassis, and industrial computer, ensuring coordinated operation of all system components. The entire system forms a closed-loop process of perception, decision-making, and execution, possessing high precision, strong adaptability, and good scalability, making it suitable for complex and dynamic warehousing and sorting environments.

[0158] In a preferred embodiment, the image processing module further includes: a convolutional neural network unit for corner projection prediction and cargo recognition; a point cloud processing unit for depth image alignment, edge detection, and point cloud segmentation; a pose calculation unit for solving the pose of the target object using the PnP algorithm; and a grasping point correction unit for performing weighted fusion calculation.

[0159] The image processing module is the core of the system's accurate recognition and positioning, comprising multiple functional units working collaboratively. First, the convolutional neural network unit analyzes color images, automatically identifying the type of goods and predicting the location of their key corner points in the image, providing 2D clues for subsequent 3D positioning. Simultaneously, the point cloud processing unit utilizes depth images acquired by a depth camera, performing multi-frame alignment and denoising, and extracting the 3D point cloud data of the target goods through edge detection and region segmentation techniques, accurately separating individual items and avoiding interference from the surrounding environment. Based on this, the pose calculation unit matches the 2D corner points predicted by the network with known 3D model points, combining camera intrinsics and employing the PnP algorithm to calculate the precise position and orientation of the goods in space. Finally, the grasping point correction unit further integrates the neural network prediction results with the geometric center obtained from point cloud segmentation, optimizing the final grasping point coordinates through a weighted approach, effectively suppressing errors from a single data source and improving the stability and reliability of positioning. The entire image processing flow starts from raw data, refining information step by step, achieving a crucial leap from "seeing" to "accurate positioning."

[0160] In a preferred embodiment, the robotic arm control module further includes:

[0161] The inverse kinematics unit is used to convert pose information into joint control commands;

[0162] The end effector control unit is used to control the adsorption and release of the pneumatic suction cup;

[0163] The path planning unit is used for obstacle avoidance and optimal path generation.

[0164] The robotic arm control module is responsible for converting the recognition results into actual grasping actions. Its internal units work together to complete the entire process from planning to execution. First, the inverse kinematics calculation unit receives the target object's pose information from the image processing module and, combined with the robotic arm's own structural parameters, calculates the rotation angles of each joint required to reach that position and posture, thus generating precise joint control commands. Subsequently, the path planning unit plans a safe, smooth, and efficient motion trajectory based on the robotic arm's starting position, the surrounding environment, and obstacle distribution, ensuring no collisions occur during movement and prioritizing paths with low energy consumption and short processing time. Upon reaching the target point, the end effector control unit activates, adjusting the vacuum pressure of the pneumatic suction cup according to the material of the cargo surface, and controlling the solenoid valve to achieve suction or release actions, ensuring a firm grip while avoiding damage to fragile packaging. The entire control process is achieved through close cooperation between the units, responding in real time under the unified scheduling of the ROS system, realizing intelligent, precise, and safe robotic arm movements.

[0165] In a preferred embodiment, the system further includes a database management module for storing cargo model files, sorting task information, and identification logs, and supports data querying and updating using SQL commands.

[0166] The system also features a database management module for unified storage and management of various key data, ensuring traceability and maintainability of the sorting process. This module stores 3D model files of goods, providing a baseline template for visual recognition and pose estimation; it also records sorting task information for each batch, including target shelf, goods type, and processing status, supporting task tracking and playback. Furthermore, recognition results, grab success rates, and anomaly logs generated during system operation are stored in the database in real time for subsequent analysis and optimization. All data is organized in a structured manner, supporting rapid querying, updating, and exporting via standard SQL commands. For example, it allows retrieving historical processing records for a specific type of goods or adding model data for a new packaging type. This module interacts with other functional units of the system via ROS or network interfaces, ensuring data consistency while enhancing the system's flexibility and scalability, providing data support for the long-term stable operation of intelligent warehousing.

[0167] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A convolutional neural network visual sorting method for intelligent warehousing, characterized in that, Includes the following steps: Obtain the order information to be sorted within a preset time period, and divide and reorganize the orders according to the product category and demand to generate multiple sorting sequence sets; Based on the demand and sorting priority of goods in the sorting sequence, sorting levels are set and assigned to different sorting areas, and corresponding sorting equipment tasks are configured. Collect image information of goods to be sorted, extract regions of interest after image preprocessing, and obtain image features and point cloud features; A cargo recognition and pose estimation model based on convolutional neural networks is constructed. The model is jointly trained by combining image features and point cloud features to output the cargo category and spatial pose. Based on the recognition results and pose information, the robotic arm and end effector are controlled to complete the grasping and sorting of goods; The construction of a cargo recognition and pose estimation model based on a convolutional neural network, which combines image features and point cloud features for joint training, outputs the cargo category and spatial pose, specifically including: An improved YOLO-V2 network structure is adopted, with a 416×416×3 color image as input and the output being the two-dimensional projected coordinates of multiple corner points and their confidence scores. Semi-automatic annotation was performed using Apriltags visual labeling tools to generate a high-precision training dataset. The pose of the target object relative to the camera coordinate system is solved by combining the known 3D corner coordinates with the 2D projection points predicted by the network using the PnP algorithm. Nonmaximum suppression and point cloud clustering algorithms are introduced to filter and correct the predicted pose, thereby improving the pose estimation accuracy. When obtaining the final captured point coordinates, a weighted fusion method is used to correct the network prediction bias. The expression is as follows: ； in, The final capture point coordinates; The 3D point coordinates obtained by backprojecting the predicted corner points from the network; The coordinates of the dense center of the point cloud obtained from point cloud segmentation; The fusion weighting coefficient has a range of values. ; Before performing the grabbing action, a surface geometric stability score for the candidate grabbing region is calculated to select the optimal grabbing point; the stability score is calculated using the following formula: ； in, The surface geometric stability score is given, with a value range of [value range missing]. A larger value indicates that the area is more suitable for stable data capture. This represents the number of point cloud points sampled within the candidate capture area; For the first Unit normal vector of each point cloud point; This is the average vector of the normal vectors of all point cloud points within the candidate region; It is the Euclidean norm of the vector difference, reflecting the degree of deviation of the local normal. For the first The curvature value of the neighborhood of a point in the point cloud; This is the curvature penalty coefficient, used to adjust the intensity of the curvature's influence on the score, with a value range of [value missing]. ; This is the exponential decay factor for the effect of curvature; when If the candidate grab area is determined to meet the geometric stability requirements, the grab operation is allowed. After performing the crawling operation, if a crawling failure is detected, an adaptive retry mechanism is initiated to calculate the predicted crawling success rate of the current candidate crawling points and adjust the strategy accordingly. The expression for the crawling success rate prediction model is: ； in, This represents the predicted success rate of crawling, with a value range of [0, 1]. Indicates the coefficient of kinetic friction of the material on the surface of the goods; This indicates the effective contact area between the pneumatic suction cup and the surface of the goods. Indicates the area gain coefficient; like If the success rate is deemed high, a maximum of two parameter fine-tuning retries are allowed. like The system prompts for manual review or switching to an alternative crawling strategy. like If the capture point is not found, the visual recognition process is restarted to find a new candidate region.

2. The convolutional neural network visual sorting method for intelligent warehousing according to claim 1, characterized in that, The process of obtaining order information to be sorted within a preset time period, segmenting and reorganizing the orders according to the product category and demand volume, and generating multiple sorting sequence sets specifically includes: Extract product category keywords and quantity information from the order to generate a product data sequence; Calculate the similarity between different product data sequences, and merge sequences with similarity higher than a preset threshold into the same sorting sequence set; The sorting sequence is optimized by sorting the goods according to their average demand time and optimizing the sorting path and task scheduling order.

3. The convolutional neural network visual sorting method for intelligent warehousing according to claim 1, characterized in that, Images of goods to be sorted are acquired using a depth camera. After image preprocessing, regions of interest are extracted to obtain image features and point cloud features, specifically including: Color and depth images were acquired using a structured light depth camera; The image is denoised by median filtering and spatial nearest neighbor filtering, and the depth image and color image are aligned. The corner projection regions in the image are extracted based on the convolutional neural network, and the three-dimensional corner reconstruction is performed by combining the point cloud information. By using edge detection and point cloud segmentation algorithms, the dense center of the point cloud of the target object is extracted to obtain candidate grabbing regions.

4. The convolutional neural network visual sorting method for intelligent warehousing according to claim 1, characterized in that, Before performing the capture, it is necessary to verify the consistency between the network predicted points and the point cloud center. The criterion is as follows: ； in, The distance between the two points is Euclidean distance; when the criterion is met, the pose estimation result is deemed reliable, and the grabbing operation is allowed.

5. The convolutional neural network visual sorting method for intelligent warehousing according to claim 1, characterized in that, The process of controlling the robotic arm and end effector to grasp and sort goods based on the recognition results and pose information specifically includes: The target object's pose information is published to the robotic arm control node via the ROS system; joint control commands are generated through inverse kinematics calculation to drive the robotic arm to move to the gripping point; the pneumatic suction cup gripper is controlled to complete the adsorption and release of goods; and autonomous navigation and positioning between multiple shelves are achieved by combining with the omnidirectional mobile chassis.

6. A convolutional neural network visual sorting system for intelligent warehousing, characterized in that, The method employs a convolutional neural network-based visual sorting approach for intelligent warehousing, as described in any one of claims 1-5, comprising: Depth camera module, used to acquire color and depth images of goods to be sorted; The image processing module is used to extract image features and point cloud features, and to perform pose estimation; The robotic arm control module is used to control the robotic arm to complete the grasping operation based on the pose information; The mobile chassis module is used to enable the sorting platform to move autonomously between multiple shelves; The ROS communication module is used for data interaction between various hardware devices and software nodes.

7. A convolutional neural network visual sorting system for intelligent warehousing according to claim 6, characterized in that, The image processing module further includes: a convolutional neural network unit for corner projection prediction and cargo recognition; a point cloud processing unit for depth image alignment, edge detection, and point cloud segmentation; a pose calculation unit for solving the pose of the target object using the PnP algorithm; and a grasping point correction unit for performing weighted fusion calculation.

Images