Intelligent warehousing goods automatic screening method and device based on machine vision

Abstract

This invention discloses an automatic sorting method and apparatus for intelligent warehouse goods based on machine vision, relating to the field of intelligent warehousing technology. The method includes: acquiring the original image and depth map of the goods, generating registered standard RGB-D image pairs; inputting the registered standard RGB-D image pairs into a target detection and segmentation model to generate a bounding box set and a pixel-level mask matrix set; performing region cropping and projection to extract and generate a multi-dimensional joint feature vector; inputting the multi-dimensional joint feature vector into a defect classifier to generate a comprehensive evaluation result label; outputting a static screening level label based on the category information of the bounding box set and the comprehensive evaluation result label, and generating sorting control instructions in conjunction with a dynamic tracking algorithm to drive a pneumatic sorting mechanism to complete physical sorting. This invention solves the problems of low image registration accuracy, poor model hyperparameter optimization efficiency, single defect evaluation dimension, and high dynamic sorting bias rate in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of intelligent warehousing technology, and in particular to an intelligent warehousing automatic sorting method and apparatus based on machine vision. Background Technology

[0002] In modern intelligent warehousing and logistics sorting systems, automated product sorting is a key element in improving overall efficiency. Traditional sorting methods rely heavily on manual visual inspection or simple photoelectric sensors, which suffer from low efficiency, fatigue, and inability to identify minute defects and deformations on the product surface. In recent years, machine vision-based detection technology has been gradually introduced into sorting lines, using deep learning models to identify and locate product categories.

[0003] However, existing technologies still have the following technical shortcomings in practical applications: 1) Insufficient image acquisition and fusion accuracy: The production line environment is complex and the lighting is variable. A single RGB camera is prone to failure when the color of the goods is similar to the background. The simple stitching of traditional RGB and depth maps has problems such as large registration error and blurred edges, which leads to inaccurate subsequent segmentation.

[0004] 2) Weak generalization ability and difficulty in parameter tuning: Mainstream object detection and segmentation models (such as the YOLO series) contain a large number of hyperparameters. Traditional parameter tuning methods that rely on manual experience or grid search are time-consuming and difficult to find the global optimal solution, resulting in missed detections or missegments in complex pipeline scenarios.

[0005] 3) Single defect judgment features: Existing visual sorting is mostly limited to the "category recognition" level, lacking the joint extraction of multi-dimensional features such as packaging text, surface damage, and three-dimensional deformation, which makes it difficult to meet the high-standard warehouse quality inspection requirements.

[0006] 4) Poor dynamic sorting coordination: There is often a time delay between the visual processing results and the physical pneumatic sorting mechanism. The lack of accurate dynamic tracking and delay compensation mechanism leads to frequent "off-target" or "missed sorting" phenomena on high-speed assembly lines. Summary of the Invention

[0007] This invention provides a machine vision-based intelligent warehouse goods automatic sorting method and device. This invention solves the problems of low image registration accuracy, poor model hyperparameter optimization efficiency, single defect assessment dimension, and high dynamic sorting bias rate in existing technologies.

[0008] In a first aspect, embodiments of the present invention provide an intelligent warehouse goods automatic sorting method based on machine vision, the method comprising: The original images and depth maps of the goods on the production line are acquired by machine vision sensors, and noise reduction and distortion correction are performed to generate registered standard RGB-D image pairs. The registered standard RGB-D image pairs are input into a target detection and segmentation model based on deep learning and optimized by an improved jackal swarm optimization algorithm to generate a set of bounding boxes containing class probabilities and a set of pixel-level mask matrices. Based on the set of pixel-level mask matrices, region cropping and projection are performed on the registered standard RGB-D image pairs to extract and generate a multidimensional joint feature vector for each item. The multidimensional joint feature vector is input into a metric learning-based defect classifier to generate a comprehensive evaluation result label that includes defect type and deformation index. Based on the category information of the bounding box set and the comprehensive evaluation result label, a static screening level label is output, and a sorting control instruction is generated in combination with the dynamic tracking algorithm to drive the pneumatic sorting mechanism to complete the physical sorting of warehouse goods.

[0009] The technical solution provided in this application has at least the following beneficial effects: Adaptive median filtering and bilateral filtering are used for targeted denoising of RGB and depth maps respectively. Combined with the extrinsic reprojection method of Zhang Zhengyou calibration, sensor noise and lens distortion are effectively eliminated, achieving pixel-level accurate registration. This provides high-quality input for the dual-stream network feature extraction of subsequent object detection and segmentation models. An adaptive weighted fusion mechanism is introduced, which can automatically adjust the weights of 2D semantic and 3D structural features based on the similarity between the goods and the background, effectively overcoming the failure problem caused by illumination changes and similar colors. An improved jackal swarm optimization algorithm is proposed, introducing Logistic chaotic mapping initialization to improve population diversity. Particle swarm optimization mechanism and nonlinear convergence factor are deeply integrated in the search, encirclement, and attack stages. This mechanism significantly suppresses the premature convergence defect of the original algorithm, achieving efficient and accurate global optimization of model hyperparameters and improving the robustness of detection and segmentation under complex pipelines. In the improved YOLOv8-seg object detection and segmentation model, a dual-stream backbone of RGB and Depth streams is set up. Adaptive fusion weights are generated through global average pooling and Sigmoid. When the product color is confused with the background, the network can automatically increase the weight of deep structural features, which greatly enhances the anti-interference ability of instance segmentation. Based on the instance segmentation mask, a multi-dimensional joint feature vector that integrates "appearance feature vector, text semantic feature vector, and geometric morphological parameter vector" is extracted. Combined with Mahalanobis distance metric learning, a comprehensive evaluation of scratches, stains, text errors, and three-dimensional physical deformations (such as dents and bulges) is achieved. By coordinating the physical coordinate motion trajectory with timestamps and the pneumatic push rod baseline, precise delay trigger time calculation is introduced to eliminate the position deviation caused by the speed fluctuation of the production line, realizing a closed loop of "millisecond-level visual perception - millimeter-level physical sorting".

[0010] In one optional implementation, the original image and depth map of the goods on the production line are acquired through a machine vision sensor, denoising and distortion correction are performed, and a registered standard RGB-D image pair is generated, including: Use industrial cameras in machine vision sensors to acquire raw RGB images of goods on the production line; Simultaneously use the 3D structured light camera in the machine vision sensor to acquire the corresponding depth map; The original RGB image is subjected to adaptive median filtering for noise reduction, and lens distortion correction is performed using the pre-stored camera intrinsic parameter matrix and distortion coefficients to obtain the distorted RGB image. A bilateral filter is applied to the depth map to smooth out invalid depth values, resulting in a filtered depth map. Based on the extrinsic rotation matrix and translation vector between the industrial camera and the 3D structured light camera obtained by Zhang Zhengyou's calibration method, pixel-level reprojection alignment is performed on the distorted RGB image and the filtered depth image to generate a registered standard RGB-D image pair.

[0011] In one optional implementation, the method for constructing and optimizing the target detection and segmentation model includes: An initial object detection and segmentation model is constructed using the improved YOLOv8-seg algorithm based on deep learning. The object detection and segmentation model includes a two-stream feature extraction backbone, a neck network, and a decoupling head. The hyperparameter vectors of the initial target detection and segmentation model are encoded into the position vectors of individuals in the improved jackal swarm optimization algorithm, and the validation set loss function is used as the fitness function. Based on the fitness function, an improved jackal swarm optimization algorithm is used to iteratively optimize the hyperparameter vector to obtain the optimal hyperparameter vector; Based on the optimal hyperparameter vector, the hyperparameters of the initial target detection and segmentation model are optimized to obtain the final target detection and segmentation model optimized by the improved jackal swarm optimization algorithm.

[0012] In one alternative implementation, based on the fitness function, an improved jackal swarm optimization algorithm is used to iteratively optimize the hyperparameter vector to obtain the optimal hyperparameter vector, including: The chaotic sequence is generated using the Logistic mapping and then mapped to the solution space of individuals in the improved jackal swarm optimization algorithm to obtain the initial population including several initial individuals and the initial velocity of each individual. Based on the fitness function, the initial population and its initial velocity, a convergence factor and PSO mechanism are introduced, and an improved jackal swarm optimization algorithm is used to iteratively search for the optimal individual in the solution space with the best fitness value. The optimal hyperparameter vector is obtained by decoding the position vector of the globally optimal individual.

[0013] In one alternative implementation, based on the fitness function and the initial population, a convergence factor and a PSO mechanism are introduced. An improved jackal swarm optimization algorithm is used to iteratively search the solution space to obtain the globally optimal individual with the best fitness value, including: Based on the fitness function, the fitness of each individual in the initial population or the updated population of the previous iteration is calculated, and the optimal position of each individual and the global optimal individual of the population are determined according to the fitness. Based on the number of population members, vocal signals, and individual speeds in the current iteration, a convergence factor and a PSO mechanism are introduced. An improved jackal population optimization algorithm is used to perform a three-stage independent position update on the initial population or the updated population from the previous iteration to obtain the updated population for the current iteration. Repeatedly update the position of the population. When the current iteration reaches the maximum number of iterations or the fitness value of the globally optimal individual meets the requirements, terminate the iterative update of the population and output the final globally optimal individual.

[0014] In one optional implementation, the registered standard RGB-D image pairs are input into a target detection and segmentation model built based on deep learning and optimized by an improved jackal swarm optimization algorithm, generating a set of bounding boxes containing class probabilities and a set of pixel-level mask matrices, including: The registered standard RGB-D image pairs are input into a target detection and segmentation model based on deep learning and optimized by an improved jackal swarm optimization algorithm; Using the RGB flow feature extraction branch of the dual-flow feature extraction backbone in the object detection and segmentation model, two-dimensional semantic feature maps of the RGB flow in standard RGB-D image pairs are extracted. The three-dimensional structural feature map of the Depth flow in the standard RGB-D image pair is extracted using the Depth flow feature extraction branch of the dual-flow feature extraction backbone in the object detection and segmentation model. An adaptive weighted fusion mechanism is used to merge the features of the two-dimensional semantic feature map and the three-dimensional structural feature map to obtain the fused feature map. Using a neck network, feature enhancement is performed on the fused feature maps to obtain a set of enhanced feature maps at three scales; The enhanced feature map set is input into the classification and regression branches in the decoupling head to generate a set of bounding boxes containing class probabilities, coordinates, and confidence scores. The enhanced feature map set is input into the mask branch in the decoupling head to generate a global prototype vector matrix; For each item in the bounding box set, extract the multi-dimensional mask coefficient vector corresponding to its mask branch, and restore the mask of the item at low resolution. The mask is upsampled to the same size as the RGB stream using bilinear interpolation, then thresholded and binarized to generate a binary pixel-level mask matrix. Finally, all bounding boxes in the bounding box set are traversed to obtain a set of pixel-level mask matrices.

[0015] In one optional implementation, based on the pixel-level mask matrix set, region cropping and projection are performed on the registered standard RGB-D image pair to extract and generate a multi-dimensional joint feature vector for each item, including: Extract the binary pixel-level mask matrix of each item from the set of pixel-level mask matrices, perform element-wise multiplication with the RGB stream in the registered standard RGB-D image of the item, set the background pixels to zero, and obtain a pure foreground image; The pure foreground image is input into the pre-trained ResNet18 network to extract the corresponding appearance feature vector. The DBNet++ algorithm is then run on the pure foreground image to obtain the text bounding box. Finally, the text is encoded into the corresponding text semantic feature vector through the lightweight BERT model. Based on the binary pixel-level mask matrix corresponding to the product, the point cloud set of the corresponding region is extracted from the RGB stream of the registered standard RGB-D image. The aspect ratio of the minimum bounding rectangle and the standard deviation of the surface curvature of the point cloud set are calculated to construct the geometric morphological parameter vector. The apparent feature vector, textual semantic feature vector, and geometric morphological parameter vector of the same product are concatenated along the channel dimension, and then nonlinear dimensionality reduction mapping is performed through a fully connected layer to obtain the multidimensional joint feature vector of the product.

[0016] In one optional implementation, the multidimensional joint feature vector is input into a metric learning-based defect classifier to generate a comprehensive evaluation result label containing defect type and deformation index, including: The multidimensional joint feature vector is input into a defect classifier based on metric learning, and the Mahalanobis distance set between the multidimensional joint feature vector and a pre-constructed prototype feature vector library of several defects is calculated. Read the geometric morphology parameter vector in the multidimensional joint feature vector. If the standard deviation of surface curvature in the geometric morphology parameter vector is greater than the preset deformation threshold, then generate a Boolean deformation index of physical damage. If the minimum value in the Mahalanobis distance set is less than the defect threshold, the corresponding defect type code is output, and the deformation index is read to obtain a comprehensive evaluation result label containing the defect type and deformation index.

[0017] In one optional implementation, static screening level labels are output based on the category information of the bounding box set and the comprehensive evaluation result labels. These labels are then combined with a dynamic tracking algorithm to generate sorting control instructions, driving the pneumatic sorting mechanism to complete the physical sorting of warehouse goods, including: The category information in the bounding box set and the comprehensive evaluation result label are input into the decision tree to generate static screening level labels. Based on the bounding box set, a multi-object tracking algorithm is used to generate physical coordinate motion trajectories with timestamps. If the horizontal vector in the physical coordinate motion trajectory reaches the preset pneumatic push rod trigger baseline, read the current static screening level label and calculate the precise delay trigger time; After a precise delay, an execution command containing the push rod number and trigger pulse width is sent to the pneumatic sorting mechanism. The pneumatic mechanism then operates to complete the physical sorting of the stored goods.

[0018] Secondly, embodiments of the present invention provide an intelligent warehouse goods automatic sorting device based on machine vision, used to implement an intelligent warehouse goods automatic sorting method, the device comprising: The machine vision unit is used to acquire the original images and depth maps of the goods on the production line through machine vision sensors, perform noise reduction and distortion correction, and generate registered standard RGB-D image pairs. The target detection and segmentation unit is used to input the registered standard RGB-D image pair into the target detection and segmentation model based on deep learning and optimized by the improved jackal swarm optimization algorithm, and generate a set of bounding boxes containing class probabilities and a set of pixel-level mask matrices. The region cropping and projection unit is used to perform region cropping and projection on the registered standard RGB-D image pair according to the pixel-level mask matrix set, and extract and generate a multi-dimensional joint feature vector for each product. The defect classification unit is used to input the multidimensional joint feature vector into a metric learning-based defect classifier to generate a comprehensive evaluation result label that includes defect type and deformation index. The automatic screening execution unit is used to output static screening level labels based on the category information of the bounding box set and the comprehensive evaluation result labels, and to generate sorting control instructions in combination with the dynamic tracking algorithm to drive the pneumatic sorting mechanism to complete the physical sorting of warehouse goods.

[0019] A third aspect of this invention provides an electronic device, which includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps of an intelligent warehouse goods automatic sorting method based on machine vision, as provided in an embodiment of the present invention. Figure 3 This is a functional unit diagram of an intelligent warehouse goods automatic sorting device based on machine vision provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0023] The present invention will be further described below with reference to the accompanying drawings.

[0024] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0025] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0026] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0027] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for an intelligent warehouse goods automatic sorting device based on machine vision.

[0028] exist Figure 1In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the machine vision-based intelligent warehouse goods automatic sorting device stored in the memory 1005 through the processor 1001, and executes the machine vision-based intelligent warehouse goods automatic sorting method provided in the embodiment of the present invention.

[0029] Reference Figure 2 The present invention provides an intelligent warehouse goods automatic sorting method based on machine vision, the method comprising: S201: Acquire the original images and depth maps of goods on the production line through machine vision sensors, perform noise reduction and distortion correction, and generate registered standard RGB-D image pairs; S202: Input the registered standard RGB-D image pair into a target detection and segmentation model based on deep learning and optimized by an improved jackal swarm optimization algorithm to generate a set of bounding boxes containing class probabilities and a set of pixel-level mask matrices; S203: Based on the set of pixel-level mask matrices, perform region cropping and projection on the registered standard RGB-D image pair to extract and generate a multidimensional joint feature vector for each item; S204: Input the multidimensional joint feature vector into a metric learning-based defect classifier to generate a comprehensive evaluation result label that includes defect type and deformation index; S205: Based on the category information of the bounding box set and the comprehensive evaluation result label, output static screening level label, and generate sorting control instructions in combination with dynamic tracking algorithm to drive the pneumatic sorting mechanism to complete the physical sorting of warehouse goods.

[0030] The technical solution provided in this application has at least the following beneficial effects: Adaptive median filtering and bilateral filtering are used for targeted denoising of RGB and depth maps respectively. Combined with the extrinsic reprojection method of Zhang Zhengyou calibration, sensor noise and lens distortion are effectively eliminated, achieving pixel-level accurate registration. This provides high-quality input for the dual-stream network feature extraction of subsequent object detection and segmentation models. An adaptive weighted fusion mechanism is introduced, which can automatically adjust the weights of 2D semantic and 3D structural features based on the similarity between the goods and the background, effectively overcoming the failure problem caused by illumination changes and similar colors. An improved jackal swarm optimization algorithm is proposed, introducing Logistic chaotic mapping initialization to improve population diversity. Particle swarm optimization mechanism and nonlinear convergence factor are deeply integrated in the search, encirclement, and attack stages. This mechanism significantly suppresses the premature convergence defect of the original algorithm, achieving efficient and accurate global optimization of model hyperparameters and improving the robustness of detection and segmentation under complex pipelines. In the improved YOLOv8-seg object detection and segmentation model, a dual-stream backbone of RGB and Depth streams is set up. Adaptive fusion weights are generated through global average pooling and Sigmoid. When the product color is confused with the background, the network can automatically increase the weight of deep structural features, which greatly enhances the anti-interference ability of instance segmentation. Based on the instance segmentation mask, a multi-dimensional joint feature vector that integrates "appearance feature vector, text semantic feature vector, and geometric morphological parameter vector" is extracted. Combined with Mahalanobis distance metric learning, a comprehensive evaluation of scratches, stains, text errors, and three-dimensional physical deformations (such as dents and bulges) is achieved. By coordinating the physical coordinate motion trajectory with timestamps and the pneumatic push rod baseline, precise delay trigger time calculation is introduced to eliminate the position deviation caused by the speed fluctuation of the production line, realizing a closed loop of "millisecond-level visual perception - millimeter-level physical sorting".

[0031] In one optional implementation, the original image and depth map of the goods on the production line are acquired through a machine vision sensor, denoising and distortion correction are performed, and a registered standard RGB-D image pair is generated, including: S2011: Use an industrial camera in a machine vision sensor to acquire raw RGB images of goods on the production line; S2012: Simultaneously use the 3D structured light camera in the machine vision sensor to acquire the corresponding depth map; In this embodiment, to eliminate the "ghosting" caused by the assembly line motion and the spatial misalignment between the RGB and depth maps, the industrial camera and the 3D structured light camera use a hardware trigger line for hard synchronization. A square wave signal of a fixed frequency is output by the programmable logic controller (e.g., exposure time set to 1ms, trigger frequency 30Hz) to ensure that the RGB image and depth map acquired at the same time are absolutely aligned in timestamps. S2013: Perform adaptive median filtering to denoise the original RGB image, and use the pre-stored camera intrinsic parameter matrix and distortion coefficients to perform lens distortion correction to obtain the distorted RGB image. In this embodiment, an adaptive median filter is used to address isolated noise points caused by uneven illumination in the production line. This algorithm dynamically calculates the grayscale histogram of the current pixel's neighborhood and performs median replacement only when impulse noise is detected; otherwise, the original pixel value is retained. This perfectly preserves the edge texture of the product packaging and the outline of the printed fonts while denoising. The distortion correction process is based on the Brown-Conrady distortion model and utilizes the pre-calibrated radial and tangential distortion coefficients to achieve pixel-level fast resampling correction through an inverse mapping lookup table. S2014: Apply bilateral filtering to the depth map to smooth and fill invalid depth values, resulting in a filtered depth map; In this embodiment, when 3D structured light is used to photograph highly reflective materials (such as transparent plastic bags and laminated cardboard boxes) or pure black light-absorbing materials, invalid depth holes will be generated. This embodiment uses a bilateral filter with spatial-domain bidirectional filtering characteristics. Its core is to use the spatial Euclidean distance between pixels and the difference in depth values ​​to jointly construct a weight kernel function. While filling the hole depth, it strictly suppresses the depth smoothing across the physical edge of the goods to prevent "edge blurring". S2015: Based on the external parameter rotation matrix and translation vector between the industrial camera and the 3D structured light camera obtained by Zhang Zhengyou's calibration method, pixel-level reprojection alignment is performed on the distorted RGB image and the filtered depth image to generate a registered standard RGB-D image pair. In this embodiment, based on Zhang Zhengyou's calibration method, a rigid transformation matrix (containing a 3×3 rotation matrix and a 3×1 translation vector) from the industrial camera coordinate system to the 3D structured light camera coordinate system is derived. Each effective pixel in the depth map is inversely projected to a point in three-dimensional physical space, and then transformed to the industrial camera coordinate system using the rotation matrix and translation vector. Finally, it is reprojected to the two-dimensional pixel plane through the intrinsic parameter matrix of the industrial camera. Through this rigorous geometric closed loop of "2D-3D-2D", the misalignment caused by the non-parallelism of the optical axes of the two cameras and parallax is eliminated, and the registration accuracy is controlled at the sub-pixel level (error < 0.5 pixels).

[0032] In one optional implementation, the method for constructing and optimizing the target detection and segmentation model includes: A-1: Using the improved YOLOv8-seg algorithm of deep learning, an initial object detection and segmentation model is constructed, which includes a two-stream feature extraction backbone, a neck network, and a decoupling head; A-2: Encode the hyperparameter vectors of the initial target detection and segmentation model into the position vectors of individuals in the improved jackal swarm optimization algorithm, and use the validation set loss function as the fitness function; In this embodiment, the hyperparameter vector includes key hyperparameters that have a significant impact on YOLOv8-seg, such as the initial learning rate, the final learning rate, the momentum parameter, the weight decay coefficient, the Mosaic data augmentation probability, and the weight ratios of the box loss, classification loss, and DFL loss in the loss function. A-3: Based on the fitness function, an improved jackal swarm optimization algorithm is used to iteratively optimize the hyperparameter vector to obtain the optimal hyperparameter vector; A-4: Based on the optimal hyperparameter vector, optimize the hyperparameters of the initial target detection and segmentation model to obtain the final target detection and segmentation model optimized by the improved jackal swarm optimization algorithm.

[0033] In one alternative implementation, based on the fitness function, an improved jackal swarm optimization algorithm is used to iteratively optimize the hyperparameter vector to obtain the optimal hyperparameter vector, including: A-3-1: Use the Logistic mapping to generate a chaotic sequence, and map the chaotic sequence to the solution space of individuals in the improved jackal swarm optimization algorithm to obtain the initial population including several initial individuals and the initial velocity of each individual, as shown in the formula: In the formula, For the initial group of individuals, the first i An initial individual; For the first i One chaotic variable; For the upper and lower bounds of the multidimensional solution space; i For individual indicators; t This represents the current iteration number; In the formula, For the initial population, the first i The initial velocity of each initial individual; A random number in the interval (0,1); This represents the maximum speed. A-3-2: Based on the fitness function, the initial population and its initial velocity, a convergence factor and a particle swarm optimization (PSO) mechanism are introduced. An improved jackal swarm optimization algorithm is used to iteratively search for the optimal individual in the solution space and obtain the globally optimal individual with the best fitness value. A-3-3: Decode the position vector of the globally optimal individual to obtain the optimal hyperparameter vector.

[0034] In one alternative implementation, based on the fitness function and the initial population, a convergence factor and a PSO mechanism are introduced. An improved jackal swarm optimization algorithm is used to iteratively search the solution space to obtain the globally optimal individual with the best fitness value, including: A-3-2-1: Based on the fitness function, calculate the fitness of each individual in the initial population or the updated population of the previous iteration, and determine the optimal position of each individual and the global optimal individual of the population based on the fitness. A-3-2-2: Based on the current population size, vocalization signals, and individual velocities, a convergence factor and PSO mechanism are introduced. An improved jackal population optimization algorithm is used to perform a three-stage independent position update on the initial population or the population updated in the previous iteration, resulting in the updated population for the current iteration, including: A-3-2-2-1: Extract the number of population members in the current iteration and randomly generate a vocal signal as the trigger condition for the three-stage independent position update; A-3-2-2-2: If the vocal signal is less than the signal threshold and the population size is less than the population size threshold, the search phase is triggered. Based on the velocity of individuals in the previous iteration, a convergence factor and a PSO mechanism are introduced to perform a global search on the initial population or the updated population from the previous iteration, obtaining the updated population for the current iteration. The formula is: In the formula, For the first t+ The first iteration of the updated population i The rate at which each updated individual is updated; For the first t In the population of the second iteration i The speed of each individual, in the first iteration The initial velocity of the individual; For the first t The iteration of the ... i The historical best position of each individual; Accelerate one's own cognition; The acceleration coefficient of social cognition; A random number in the interval [0,1]. For the first t The globally optimal individual in the next iteration; For the first t The convergence factor of the next iteration; For the first phase of the search t+ The first iteration of the updated population i A newer individual; For the firstt The iteration of the ... i Each updated individual, in the initial iteration, For the initial individual; t This is an indicator of the number of iterations. These are the maximum and minimum values ​​of the convergence factor; This represents the maximum number of iterations. A-3-2-2-3: If the emitted signal is less than the signal threshold and the population member count is greater than or equal to the population member count threshold, then the encirclement phase is triggered. Based on the velocity of individuals in the previous iteration, a convergence factor and a PSO mechanism are introduced to perform local development on the initial population or the updated population from the previous iteration, resulting in the updated population for the current iteration. The formula is as follows: In the formula, The first phase of the encirclement t+ The first iteration of the updated population i A newer individual; A-3-2-2-4: If the vocal signal is greater than or equal to the signal threshold, the attack phase is triggered. Based on the velocity of individuals in the previous iteration, a prey size factor, a convergence factor, and a PSO mechanism are introduced to perform fine-tuning on the initial population or the updated population from the previous iteration, resulting in the updated population for the current iteration. The formula is as follows: In the formula, Prey size factor; A random number in the interval [0,1]. For the first t The iteration of the ... i The fitness values ​​of each updated individual and the globally optimal individual; If prey size factor If it is identified as large prey, the following refined development is performed to obtain the updated population for the current iteration, using the formula: In the formula, The first step in the search phase for large prey t+ The first iteration of the updated population i A newer individual; The first under large prey t In the updated population of the next iteration i The updated individual jackal pack attack locations; A random number in the interval [0,1]. If prey size factor If it is identified as a small prey, the following refined development is performed to obtain the updated population for the current iteration, using the formula: In the formula, For the search phase, the first small prey t+ The first iteration of the updated population i A newer individual; The first for small prey t In the updated population of the next iteration i The updated individual jackal pack attack locations; A random number in the interval [0,1]. A-3-2-3: Repeatedly update the position of the population. When the current iteration count reaches the maximum iteration count or the fitness value of the globally optimal individual meets the requirements, terminate the iterative update of the population and output the final globally optimal individual.

[0035] In one optional implementation, the registered standard RGB-D image pairs are input into a target detection and segmentation model built based on deep learning and optimized by an improved jackal swarm optimization algorithm, generating a set of bounding boxes containing class probabilities and a set of pixel-level mask matrices, including: S2021: Input the registered standard RGB-D image pair into a target detection and segmentation model based on deep learning and optimized by an improved jackal pack optimization algorithm; S2022: Using the RGB flow feature extraction branch of the dual-flow feature extraction backbone in the object detection and segmentation model, extract the two-dimensional semantic feature map of the RGB flow in the standard RGB-D image pair; the RGB flow feature extraction branch includes the YOLOv8 standard CBS (Conv+BatchNorm+SiLU) module and the Cross Stage Partial Bottleneck with 2 convolutions (C2f) module; S2023: Using the Depth flow feature extraction branch of the dual-flow feature extraction backbone in the object detection and segmentation model, extract the three-dimensional structural feature map of the Depth flow in the standard RGB-D image pair; the Depth flow feature extraction branch includes a lightweight C2f module; In this embodiment, the RGB stream uses the standard C2f module (which integrates cross-stage local networks and residual connections), which is good at extracting rich two-dimensional color, texture and trademark pattern features; the Depth stream, which is designed for the characteristics of depth maps that lack color but contain spatial structure, removes some channel attention mechanisms and adopts lightweight C2f to extract three-dimensional contour features while reducing the amount of computation. S2024: An adaptive weighted fusion mechanism is used to merge the features of the two-dimensional semantic feature map and the three-dimensional structural feature map to obtain a fused feature map. The fusion weights of the adaptive weighted fusion mechanism are automatically learned and generated by the network through global average pooling + Sigmoid function, so that the network automatically enhances its dependence on depth contour when the color of the goods is similar to the background. In this embodiment, the RGB feature map and the Depth feature map are concatenated along the channel dimension. The spatial dimension is then compressed using global average pooling to obtain the channel descriptor. After passing through two fully connected layers and a Sigmoid activation function, a weight vector (with values ​​from 0 to 1) of the same length as the number of channels is output. This weight vector is multiplied by the original feature map. When the color of the goods (such as a kraft paper box) is extremely similar to the background color of the conveyor belt, the network will automatically learn to suppress the weights of the RGB stream (approaching 0) and amplify the weights of the Depth stream (approaching 1) during forward propagation, thereby accurately locating the target based on the depth contour. S2025: The neck network of the Path Aggregation Network - Feature Pyramid Network (PAN-FPN) structure is used to enhance the features of the fused feature map, resulting in a set of enhanced feature maps at three scales. S2026: Input the enhanced feature map set into the classification and regression branch in the decoupling head to generate a regression on the grid points on the enhanced feature map set, predict the center point offset, width and height of the bounding box and the class probability of each category. After decoding and non-maximum suppression, a set of bounding boxes containing class probability, coordinates and confidence is generated. S2027: Input the enhanced feature map set into the mask branch in the decoupling head. While performing classification and regression, predict a multi-dimensional mask coefficient vector for each grid point on the enhanced feature map set, and generate a global prototype vector matrix through global pooling. S2028: For each item in the bounding box set, extract the multidimensional mask coefficient vector corresponding to the mask branch, and use matrix multiplication and sigmoid activation to linearly combine it with the global prototype matrix to restore the mask of the item at low resolution. S2029: Upsample the mask to the same size as the RGB stream using bilinear interpolation, perform threshold binarization to generate a binary pixel-level mask matrix, and traverse all bounding boxes in the bounding box set to obtain a set of pixel-level mask matrices.

[0036] In one optional implementation, based on the pixel-level mask matrix set, region cropping and projection are performed on the registered standard RGB-D image pair to extract and generate a multi-dimensional joint feature vector for each item, including: S2031: Extract the binary pixel-level mask matrix of each item from the set of pixel-level mask matrices, perform element-wise multiplication with the RGB stream in the registered standard RGB-D image of the item, set the background pixels to zero, and obtain a pure foreground image. In this embodiment, the generated binary mask matrix is ​​used as a logic switch and an element-wise bitwise AND operation is performed with the registered RGB image. This not only forces the pixel values ​​of complex backgrounds (conveyor belt texture, adjacent goods, dirt) to zero, but also eliminates the interference of the background on the subsequent feature extraction network. S2032: Input the pure foreground image into the pre-trained ResNet18 network to extract the corresponding appearance feature vector, and run the DBNet++ algorithm on the pure foreground image to obtain the text bounding box. Then, encode it into the corresponding text semantic feature vector through the pre-trained language model (Bidirectional Encoder Representations from Transformers, BERT) based on the Transformer architecture. In this embodiment, the pure foreground image is scaled to 224×224 and input to the pre-trained ResNet18 network. The output of its last global average pooling layer is extracted to obtain a 512-dimensional floating-point vector. This vector highly condenses the material texture, color distribution and overall visual shape of the packaging. Since warehouse goods often have production batch numbers, shelf life, or specific brand logos printed on them, these are key to determining whether the goods are qualified. We run the lightweight, dictionary-free text detection algorithm DBNet++ on a pure foreground image to obtain text line bounding boxes. After cropping, the bounding boxes are fed into a lightweight BERT model to extract the 768-dimensional hidden state vector of the [CLS] label, transforming the visual text into deep semantic logic features. S2033: Based on the binary pixel-level mask matrix corresponding to the goods, extract the point cloud set of the corresponding region from the RGB stream of the registered standard RGB-D image, calculate the aspect ratio of the minimum bounding rectangle and the standard deviation of the surface curvature of the point cloud set, and construct the geometric morphological parameter vector. In this embodiment, based on the coordinate index of the binary pixel-level mask matrix, the real physical depth value of the corresponding pixel is extracted from the registered depth map (i.e., RGB stream), and the two-dimensional coordinate axis is calculated in reverse by combining the camera intrinsic parameters to construct a local three-dimensional point cloud. Minimum bounding rectangle aspect ratio: Project the point cloud onto the XY plane and use the rotating caliper algorithm to calculate the minimum bounding rectangle. Its aspect ratio is used to describe whether the goods have been "flattened" or "tilted". Surface curvature standard deviation: Principal component analysis is used to cluster the normal vectors of local point clouds, and the rate of change of the normal vectors of adjacent points is calculated as curvature. The larger the standard deviation, the more uneven the surface of the goods (such as cardboard boxes that are damp and swollen, or local dents and damage). S2034: The apparent feature vector, textual semantic feature vector, and geometric morphological parameter vector of the same product are concatenated in the channel dimension, and non-linear dimensionality reduction mapping is performed through a fully connected layer to obtain the multi-dimensional joint feature vector of the product. In this embodiment, a 512-dimensional appearance vector, a 768-dimensional text vector, and 2-dimensional geometric parameters are concatenated into a 1282-dimensional joint vector. To avoid the "curse of dimensionality" and interference from redundant information, this vector is input into a fully connected network containing one hidden layer (256 neurons), and is supplemented with the LeakyReLU activation function and Dropout (with a dropout rate of 0.3 to prevent overfitting). Finally, it is nonlinearly mapped into a 128-dimensional multidimensional joint feature vector.

[0037] In one optional implementation, the multidimensional joint feature vector is input into a metric learning-based defect classifier to generate a comprehensive evaluation result label containing defect type and deformation index, including: S2041: Input the multidimensional joint feature vector into a defect classifier based on metric learning, and calculate the Mahalanobis distance set between the multidimensional joint feature vector and a pre-constructed prototype feature vector library of several defects; In this embodiment, several samples are collected in advance from the database for several typical defects (such as tearing, stains, color difference, and blurry printing), and their in-class center vectors are calculated to form a prototype feature vector library of several prototype feature vectors. S2042: Read the geometric morphology parameter vector in the multidimensional joint feature vector. If the standard deviation of surface curvature in the geometric morphology parameter vector is greater than the preset deformation threshold, then generate a Boolean-type deformation index of physical damage. S2043: If the minimum value in the Mahalanobis distance set is less than the defect threshold, output the corresponding defect type code and read the deformation index to obtain a comprehensive evaluation result label containing the defect type and deformation index.

[0038] In one optional implementation, static screening level labels are output based on the category information of the bounding box set and the comprehensive evaluation result labels. These labels are then combined with a dynamic tracking algorithm to generate sorting control instructions, driving the pneumatic sorting mechanism to complete the physical sorting of warehouse goods, including: S2051: Input the category information in the bounding box set and the comprehensive evaluation result label into the decision tree to generate a static screening level label. Based on the bounding box set, use a multi-object tracking algorithm to generate a physical coordinate motion trajectory with a timestamp. In this embodiment, a rule-based safety decision tree is constructed. For example, if the comprehensive label contains any defect type code, or the deformation index is physical damage, the static screening level label is output as "Reject". Otherwise, "Good Product A", "Good Product B", etc. are output according to the category information. Using a multi-target tracking algorithm (such as ByteTrack based on Kalman filtering and Hungarian algorithm matching), a unique tracking ID is assigned to each detection box in the pipeline field of view. The Kalman filter is used to predict the pixel center point coordinates in multiple consecutive frames. Combined with the intrinsic and extrinsic parameters of the camera calibration, the pixel coordinates are converted in real time to coordinates in the physical coordinate system of the conveyor belt (unit: millimeter), generating a motion trajectory queue with millisecond-level timestamps. S2052: If the horizontal vector in the physical coordinate motion trajectory reaches the preset pneumatic push rod trigger baseline, read the current static screening level label and calculate the precise delay trigger time; In this embodiment, an invisible "pneumatic push rod trigger baseline" (e.g., 50mm from the physical zero point of the push rod) is set at a fixed position above the conveyor belt. When the tracking algorithm calculates that the horizontal vector of the cargo's center of gravity reaches this baseline, the current static screening grade label is immediately locked. Due to the inherent mechanical response time of the push rod cylinder (e.g., 30ms), and the continued forward movement of the cargo, the precise delayed trigger time is calculated using the following formula: In the formula, To precisely delay the trigger time; Hit the target at the physical coordinates; This serves as the reference line for triggering the pneumatic actuator. The real-time speed of the assembly line; For mechanical response time; S2053: After a precise delay trigger time, an execution command containing the push rod number and trigger pulse width is sent to the pneumatic sorting mechanism. The pneumatic mechanism then operates to complete the physical sorting of the stored goods. In this embodiment, the internal high-precision timer waits for the precise delay trigger time. Then, the programmable logic controller or the corresponding pneumatic solenoid valve is sent an execution command. The command not only includes the action signal, but also precisely encodes the trigger pulse width according to the weight of the goods and the required push distance (such as controlling the solenoid valve to be energized for 45ms, which corresponds to the amount of air that the cylinder push rod extends to hit the center of gravity of the goods), so as to achieve millisecond-level precise physical sorting by combining "software and hardware".

[0039] This invention also provides an intelligent warehouse goods automatic sorting device 300 based on machine vision, see reference. Figure 3 The device may include the following units: The machine vision unit 301 is used to acquire the original image and depth map of the goods on the production line through the machine vision sensor, perform noise reduction and distortion correction, and generate a registered standard RGB-D image pair. The target detection and segmentation unit 302 is used to input the registered standard RGB-D image pair into the target detection and segmentation model based on deep learning and optimized by the improved jackal group optimization algorithm, and generate a set of bounding boxes containing class probabilities and a set of pixel-level mask matrices. The region cropping and projection unit 303 is used to perform region cropping and projection on the registered standard RGB-D image pair according to the pixel-level mask matrix set, and extract and generate a multi-dimensional joint feature vector for each item. The defect classification unit 304 is used to input the multidimensional joint feature vector into a metric learning-based defect classifier to generate a comprehensive evaluation result label that includes defect type and deformation index. The automatic sorting execution unit 305 is used to output static sorting level labels based on the category information of the bounding box set and the comprehensive evaluation result labels, and generate sorting control instructions in combination with the dynamic tracking algorithm to drive the pneumatic sorting mechanism to complete the physical sorting of warehouse goods.

[0040] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; The processor, when executing the program stored in the memory, implements the machine vision-based intelligent warehouse goods automatic sorting method of the present invention.

[0041] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EI) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM), or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0042] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0043] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the machine vision-based intelligent warehouse goods automatic sorting method of the present invention.

[0044] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0045] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0046] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0047] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0048] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0049] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A machine vision-based intelligent warehouse goods automatic sorting method, characterized in that, The method includes: The original images and depth maps of the goods on the production line are acquired by machine vision sensors, and noise reduction and distortion correction are performed to generate registered standard RGB-D image pairs. The registered standard RGB-D image pairs are input into a target detection and segmentation model based on deep learning and optimized by an improved jackal swarm optimization algorithm to generate a set of bounding boxes containing class probabilities and a set of pixel-level mask matrices. Based on the set of pixel-level mask matrices, region cropping and projection are performed on the registered standard RGB-D image pairs to extract and generate a multidimensional joint feature vector for each item. The multidimensional joint feature vector is input into a metric learning-based defect classifier to generate a comprehensive evaluation result label that includes defect type and deformation index. Based on the category information of the bounding box set and the comprehensive evaluation result label, a static screening level label is output, and a sorting control instruction is generated in combination with the dynamic tracking algorithm to drive the pneumatic sorting mechanism to complete the physical sorting of warehouse goods.

2. The intelligent warehouse goods automatic sorting method based on machine vision according to claim 1, characterized in that, The system uses machine vision sensors to acquire raw images and depth maps of goods on the production line, performs noise reduction and distortion correction, and generates registered standard RGB-D image pairs, including: Use industrial cameras in machine vision sensors to acquire raw RGB images of goods on the production line; Simultaneously use the 3D structured light camera in the machine vision sensor to acquire the corresponding depth map; The original RGB image is subjected to adaptive median filtering for noise reduction, and lens distortion correction is performed using the pre-stored camera intrinsic parameter matrix and distortion coefficients to obtain the distorted RGB image. A bilateral filter is applied to the depth map to smooth out invalid depth values, resulting in a filtered depth map. Based on the extrinsic rotation matrix and translation vector between the industrial camera and the 3D structured light camera obtained by Zhang Zhengyou's calibration method, pixel-level reprojection alignment is performed on the distorted RGB image and the filtered depth image to generate a registered standard RGB-D image pair.

3. The intelligent warehouse goods automatic sorting method based on machine vision according to claim 2, characterized in that, The method for constructing and optimizing the target detection and segmentation model includes: An initial object detection and segmentation model is constructed using the improved YOLOv8-seg algorithm based on deep learning. The object detection and segmentation model includes a two-stream feature extraction backbone, a neck network, and a decoupling head. The hyperparameter vectors of the initial target detection and segmentation model are encoded into the position vectors of individuals in the improved jackal swarm optimization algorithm, and the validation set loss function is used as the fitness function. Based on the fitness function, an improved jackal swarm optimization algorithm is used to iteratively optimize the hyperparameter vector to obtain the optimal hyperparameter vector; Based on the optimal hyperparameter vector, the hyperparameters of the initial target detection and segmentation model are optimized to obtain the final target detection and segmentation model optimized by the improved jackal swarm optimization algorithm.

4. The intelligent warehouse goods automatic sorting method based on machine vision according to claim 3, characterized in that, Based on the fitness function, an improved jackal swarm optimization algorithm is used to iteratively optimize the hyperparameter vector to obtain the optimal hyperparameter vector, including: The chaotic sequence is generated using the Logistic mapping and then mapped to the solution space of individuals in the improved jackal swarm optimization algorithm to obtain the initial population including several initial individuals and the initial velocity of each individual. Based on the fitness function, the initial population and its initial velocity, a convergence factor and PSO mechanism are introduced, and an improved jackal swarm optimization algorithm is used to iteratively search for the optimal individual in the solution space with the best fitness value. The optimal hyperparameter vector is obtained by decoding the position vector of the globally optimal individual.

5. The intelligent warehouse goods automatic sorting method based on machine vision according to claim 4, characterized in that, Based on the fitness function and the initial population, a convergence factor and a PSO mechanism are introduced. An improved jackal swarm optimization algorithm is used to iteratively search the solution space to obtain the globally optimal individual with the best fitness value, including: Based on the fitness function, the fitness of each individual in the initial population or the updated population of the previous iteration is calculated, and the optimal position of each individual and the global optimal individual of the population are determined according to the fitness. Based on the number of population members, vocal signals, and individual speeds in the current iteration, a convergence factor and a PSO mechanism are introduced. An improved jackal population optimization algorithm is used to perform a three-stage independent position update on the initial population or the updated population from the previous iteration to obtain the updated population for the current iteration. Repeatedly update the position of the population. When the current iteration reaches the maximum number of iterations or the fitness value of the globally optimal individual meets the requirements, terminate the iterative update of the population and output the final globally optimal individual.

6. The intelligent warehouse goods automatic sorting method based on machine vision according to claim 5, characterized in that, The registered standard RGB-D image pairs are input into a target detection and segmentation model built on deep learning and optimized by an improved jackal swarm optimization algorithm to generate a set of bounding boxes containing class probabilities and a set of pixel-level mask matrices, including: The registered standard RGB-D image pairs are input into a target detection and segmentation model based on deep learning and optimized by an improved jackal swarm optimization algorithm; Using the RGB flow feature extraction branch of the dual-flow feature extraction backbone in the object detection and segmentation model, two-dimensional semantic feature maps of the RGB flow in standard RGB-D image pairs are extracted. The three-dimensional structural feature map of the Depth flow in the standard RGB-D image pair is extracted using the Depth flow feature extraction branch of the dual-flow feature extraction backbone in the object detection and segmentation model. An adaptive weighted fusion mechanism is used to merge the features of the two-dimensional semantic feature map and the three-dimensional structural feature map to obtain the fused feature map. Using a neck network, feature enhancement is performed on the fused feature maps to obtain a set of enhanced feature maps at three scales; The enhanced feature map set is input into the classification and regression branches in the decoupling head to generate a set of bounding boxes containing class probabilities, coordinates, and confidence scores. The enhanced feature map set is input into the mask branch in the decoupling head to generate a global prototype vector matrix; For each item in the bounding box set, extract the multi-dimensional mask coefficient vector corresponding to its mask branch, and restore the mask of the item at low resolution. The mask is upsampled to the same size as the RGB stream using bilinear interpolation, then thresholded and binarized to generate a binary pixel-level mask matrix. Finally, all bounding boxes in the bounding box set are traversed to obtain a set of pixel-level mask matrices.

7. The intelligent warehouse goods automatic sorting method based on machine vision according to claim 6, characterized in that, Based on the pixel-level mask matrix set, region cropping and projection are performed on the registered standard RGB-D image pairs to extract and generate a multi-dimensional joint feature vector for each item, including: Extract the binary pixel-level mask matrix of each item from the set of pixel-level mask matrices, perform element-wise multiplication with the RGB stream in the registered standard RGB-D image of the item, set the background pixels to zero, and obtain a pure foreground image; The pure foreground image is input into the pre-trained ResNet18 network to extract the corresponding appearance feature vector. The DBNet++ algorithm is then run on the pure foreground image to obtain the text bounding box. Finally, the text is encoded into the corresponding text semantic feature vector through the lightweight BERT model. Based on the binary pixel-level mask matrix corresponding to the product, the point cloud set of the corresponding region is extracted from the RGB stream of the registered standard RGB-D image. The aspect ratio of the minimum bounding rectangle and the standard deviation of the surface curvature of the point cloud set are calculated to construct the geometric morphological parameter vector. The apparent feature vector, textual semantic feature vector, and geometric morphological parameter vector of the same product are concatenated along the channel dimension, and then nonlinear dimensionality reduction mapping is performed through a fully connected layer to obtain the multidimensional joint feature vector of the product.

8. The intelligent warehouse goods automatic sorting method based on machine vision according to claim 7, characterized in that, The multidimensional joint feature vector is input into a metric-learning-based defect classifier to generate a comprehensive evaluation result label that includes defect type and deformation index, including: The multidimensional joint feature vector is input into a defect classifier based on metric learning, and the Mahalanobis distance set between the multidimensional joint feature vector and a pre-constructed prototype feature vector library of several defects is calculated. Read the geometric morphology parameter vector in the multidimensional joint feature vector. If the standard deviation of surface curvature in the geometric morphology parameter vector is greater than the preset deformation threshold, then generate a Boolean deformation index of physical damage. If the minimum value in the Mahalanobis distance set is less than the defect threshold, the corresponding defect type code is output, and the deformation index is read to obtain a comprehensive evaluation result label containing the defect type and deformation index.

9. The intelligent warehouse goods automatic sorting method based on machine vision according to claim 8, characterized in that, Based on the category information of the bounding box set and the comprehensive evaluation result labels, static screening level labels are output, and sorting control instructions are generated in conjunction with a dynamic tracking algorithm to drive the pneumatic sorting mechanism to complete the physical sorting of warehouse goods, including: The category information in the bounding box set and the comprehensive evaluation result label are input into the decision tree to generate static screening level labels. Based on the bounding box set, a multi-object tracking algorithm is used to generate physical coordinate motion trajectories with timestamps. If the horizontal vector in the physical coordinate motion trajectory reaches the preset pneumatic push rod trigger baseline, read the current static screening level label and calculate the precise delay trigger time; After a precise delay, an execution command containing the push rod number and trigger pulse width is sent to the pneumatic sorting mechanism. The pneumatic mechanism then operates to complete the physical sorting of the stored goods.

10. A machine vision-based intelligent warehouse goods automatic sorting device, used to implement the intelligent warehouse goods automatic sorting method as described in any one of claims 1-9, characterized in that, The device includes: The machine vision unit is used to acquire the original images and depth maps of the goods on the production line through machine vision sensors, perform noise reduction and distortion correction, and generate registered standard RGB-D image pairs. The target detection and segmentation unit is used to input the registered standard RGB-D image pair into the target detection and segmentation model based on deep learning and optimized by the improved jackal swarm optimization algorithm, and generate a set of bounding boxes containing class probabilities and a set of pixel-level mask matrices. The region cropping and projection unit is used to perform region cropping and projection on the registered standard RGB-D image pair according to the pixel-level mask matrix set, and extract and generate a multi-dimensional joint feature vector for each product. The defect classification unit is used to input the multidimensional joint feature vector into a metric learning-based defect classifier to generate a comprehensive evaluation result label that includes defect type and deformation index. The automatic screening execution unit is used to output static screening level labels based on the category information of the bounding box set and the comprehensive evaluation result labels, and to generate sorting control instructions in combination with the dynamic tracking algorithm to drive the pneumatic sorting mechanism to complete the physical sorting of warehouse goods.

Images