Collaborative machine figure identification sorting method and system based on deep learning

Abstract

This invention relates to the field of robotic sorting technology, and more particularly to a collaborative robot object recognition and sorting method and system based on deep learning. The method acquires images and 3D distance sequences, utilizes a spatiotemporal feature extraction network and point cloud processing to obtain dynamic feature representations of objects and 3D environment models, respectively, and then fuses and identifies the object's identity and state. Based on the recognition results, a sorting dependency graph is constructed to plan the globally optimal sorting order, and finally, a collision-free motion trajectory is generated to control the robot to perform sorting. This invention achieves accurate object recognition and efficient, safe collaborative sorting operations in dynamic scenes.

Description

Technical Field

[0001] This invention relates to the field of robot sorting technology, and in particular to a collaborative robot object recognition and sorting method and system based on deep learning. Background Technology

[0002] In the field of industrial automation, collaborative robots are widely used in object sorting tasks. Their core function lies in accurately identifying objects in a scene and planning efficient and safe grasping operations. Traditional object recognition and sorting methods typically rely on static visual perception and pre-programmed grasping paths.

[0003] Conventional object recognition schemes are mainly based on single-frame two-dimensional color images to extract static appearance features such as color and texture of objects, achieving high static recognition accuracy. However, in dynamic and unstructured real-world sorting scenarios, objects may be in motion or occluded with each other. Relying solely on single-frame static image information makes it difficult to accurately determine the real-time motion trend and complete three-dimensional pose of objects, resulting in unstable recognition results.

[0004] In sorting planning, existing methods typically plan an independent trajectory for each object from the starting point to the grasping point and then to the placement point after object identification. The planning process is often based on the instantaneous environmental state at the current moment and uses heuristic rules or simple priority queues to determine the grasping order, such as sorting objects according to their distance from the robot or the level of recognition confidence. This local, isolated planning strategy lacks consideration for the overall task and dynamic changes in the scene.

[0005] The aforementioned conventional methods have significant drawbacks. Static single-frame recognition methods cannot effectively handle motion blur and temporal correlation information. When objects move rapidly or are briefly occluded, misidentification or tracking loss can easily occur, affecting the robustness of the sorting system. Independent, instantaneous state-based grasping sequence planning ignores the spatial occlusion relationships between objects and the subsequent reachability issues that may arise from the operation sequence, and may even result in invalid planning that cannot be executed. Summary of the Invention

[0006] This invention provides a method and system for object recognition and sorting in collaborative robots based on deep learning, which can solve the problems in the prior art.

[0007] A first aspect of the present invention provides a deep learning-based collaborative robot object recognition and sorting method, comprising:

[0008] Collect a continuous frame color image sequence of the area to be sorted and the corresponding three-dimensional spatial distance sequence;

[0009] A spatiotemporal feature extraction network is constructed to perform temporal modeling on the continuous frame color image sequence. The motion trend features of the object are captured by the inter-frame motion association module and spatiotemporally co-encoded with the static appearance features of a single frame to generate a dynamic feature representation containing motion prediction information.

[0010] The three-dimensional spatial distance sequence is reconstructed and registered using point cloud technology to generate a three-dimensional environment model of the scene to be sorted. The three-dimensional environment model includes geometric information of object surfaces and spatial occupancy information.

[0011] The dynamic feature representation and the three-dimensional environment model are input into the fusion recognition network to perform identity recognition and state determination on the objects to be sorted, and to obtain object recognition results and motion state markers.

[0012] Based on the object recognition results and motion state markers, a sorting dependency graph describing the spatial occlusion constraints and operation sequence constraints between objects is established through a collaborative task planner, and the globally optimal sorting sequence that satisfies the constraints is solved based on the sorting dependency graph.

[0013] Based on the globally optimal sorting order and the three-dimensional environment model, a collision-free motion trajectory for the collaborative robot is generated, and the collaborative robot is controlled to perform sorting operations according to the collision-free motion trajectory.

[0014] A spatiotemporal feature extraction network is constructed to perform temporal modeling on the continuous frame color image sequence. The motion trend features of objects are captured through an inter-frame motion correlation module and spatiotemporally co-encoded with single-frame static appearance features to generate a dynamic feature representation containing motion prediction information, including:

[0015] Pixel-level optical flow calculation is performed between adjacent frames in the continuous frame color image sequence. The displacement direction and displacement amplitude of each pixel are recorded to obtain a dense motion vector field. The spatial continuity analysis of the motion vector is used to identify the pixel set belonging to the same object and assign a motion entity identifier to each pixel set. The centroid trajectory curve of the pixel set corresponding to each motion entity identifier is extracted.

[0016] The centroid trajectory curve is fitted with a sliding window in the time dimension to obtain the rate of change of motion velocity and the deflection angle of motion direction, and the position distribution area of ​​each moving entity in the future is predicted to generate a motion trend feature vector carrying position prediction information.

[0017] For the current frame image in the continuous frame color image sequence, feature extraction is performed through a multi-layer convolutional neural network and a static appearance feature map is calculated. The static appearance feature map encodes the color texture and contour edge information of the object.

[0018] Based on the motion trend feature vector, the corresponding spatial region is located in the static appearance feature map, the feature response of the spatial region is amplified, and it is concatenated with the motion trend feature vector in the channel dimension and mapped to a dynamic feature representation of a unified dimension.

[0019] For the current frame image in the continuous frame color image sequence, feature extraction and static appearance feature map are performed using a multi-layer convolutional neural network. The static appearance feature map encodes the color texture and contour edge information of the object, including:

[0020] The current frame image is input into the first convolutional layer of a multi-layer convolutional neural network to perform preliminary feature extraction and obtain a shallow feature map. The shallow feature map is then separated into multiple independent feature channel response maps, each of which corresponds to a specific color mode or texture direction.

[0021] The shallow feature map is passed to the second and third convolutional layers for progressive feature abstraction. The texture composite pattern is captured by expanding the receptive field in the second convolutional layer, and the shape contour features and geometric structure features of the object are extracted in the third convolutional layer to obtain the deep feature map.

[0022] The gradient magnitude of the deep feature map is calculated to obtain the edge intensity response distribution. The edge intensity response distribution identifies the feature activation intensity of the object contour position. By setting an edge saliency threshold, edge feature regions with high activation intensity are filtered out to generate a contour edge enhancement mask.

[0023] The contour edge enhancement mask is applied to the shallow feature map to enhance the ability of color and texture information to be expressed at the edge position. At the same time, the contour edge enhancement mask is fused with the deep feature map through element-wise multiplication to highlight the contour edge features. The enhanced shallow feature map and the fused deep feature map are then spliced ​​together in the channel dimension to generate a static appearance feature map.

[0024] The three-dimensional spatial distance sequence is reconstructed and registered using point clouds to generate a three-dimensional environment model of the scene to be sorted. This three-dimensional environment model includes object surface geometry information and spatial occupancy information, including:

[0025] The three-dimensional spatial distance sequence is recombined into a structured depth image according to the scanning time sequence and angular resolution of the sensor. Bilateral filtering is performed to smooth the distance measurement noise and maintain the depth jump characteristics of the object edge. The filtered depth image is then back-projected into three-dimensional space to generate an ordered point cloud.

[0026] The ordered point cloud is subjected to point-by-point normal vector estimation. The surface normal vector direction of each point is calculated by using the ordered structure of the point cloud through the local tangent plane formed by adjacent rows and columns. The geometric discontinuity boundary of the object surface is detected according to the rate of change of the angle between the normal vector directions of adjacent points. The ordered point cloud is divided into multiple connected region point cloud segments along the geometric discontinuity boundary.

[0027] For each connected region point cloud segment, calculate its spatial envelope boundary. The minimum convex polyhedron is obtained by calculating the convex hull of the boundary points. The volume of the minimum convex polyhedron is marked as the spatial volume of the corresponding object. The shape similarity matching of connected region point cloud segments of adjacent acquisition frames is performed to establish the inter-frame correspondence.

[0028] The rigid body transformation parameters between point cloud fragments in connected regions of adjacent frames are calculated based on the inter-frame correspondence. The point cloud fragments in connected regions of multiple frames are aligned and fused to form a complete point cloud representation. The point cloud fragments are then reconstructed into a mesh to generate a triangular mesh geometric model of the object surface. The spatial volume is then combined to generate a three-dimensional environment model.

[0029] The dynamic feature representation and the three-dimensional environment model are input into a fusion recognition network to perform identity recognition and state determination on the objects to be sorted, obtaining object recognition results and motion state markers, including:

[0030] The dynamic feature is extracted as a vector difference between consecutive time steps, which is used as an inter-frame motion increment and accumulated by time window to obtain motion trajectory features. The object position in the three-dimensional environment model is matched and associated with the timestamp of the motion trajectory features and converted into spatial position features.

[0031] Calculate the correlation coefficient matrix between the motion trajectory features and the spatial location features, and perform singular value decomposition to extract the left singular vector and right singular vector corresponding to the principal singular value. Using the left singular vector and right singular vector as the basis, linear projection is performed on the motion trajectory features and spatial location features to obtain motion pattern projection coefficients and spatial pattern projection coefficients. The two sets of projection coefficients are multiplied element-wise to generate coupled features.

[0032] The coupling features are input into the identity recognition channel and the state determination channel respectively. The identity recognition channel extracts the spatial distribution characteristic components, which are then mapped to the category label space after global average pooling to obtain the object recognition result. The state determination channel extracts the temporal evolution characteristic components, which are then mapped to the state category space after temporal convolution to obtain the motion state label.

[0033] Based on the object recognition results and motion state markers, a sorting dependency graph describing spatial occlusion constraints and operation sequence constraints between objects is established using a cooperative task planner. The globally optimal sorting order satisfying the constraints is then solved based on this sorting dependency graph, including:

[0034] Based on the three-dimensional environment model, extract the triangular mesh geometric model of each object, emit grasping path rays from the current position of the end effector of the collaborative robot to the centroid of each object, calculate the number of intersecting faces of each ray with the triangular mesh geometric model of other objects, and establish occlusion constraint edges when the number of intersecting faces exceeds the occlusion determination threshold. Connect the occluded object and the occluding object in the dependency graph and label the occlusion constraint type.

[0035] For objects identified as being in motion, their historical position sequences are extracted, and acceleration and velocity coefficients are obtained by fitting a quadratic function using the least squares method. The probability density function of position prediction is calculated, and the variance of the probability density function is used as the position uncertainty. Based on the position uncertainty, an operation urgency value is assigned to the object, and operation order constraint edges are established in the dependency graph with the operation urgency value attached.

[0036] A depth-first traversal is performed on the sorting dependency graph. During the traversal, a stack of visited nodes is maintained. When an edge pointing to an existing node in the stack is encountered, a loop structure is identified. The operation urgency values ​​of all constraint edges in the loop are extracted. The constraint edge with the smallest operation urgency value is deleted. The traversal is re-executed until no loop exists. The final node access sequence is taken as the globally optimal sorting order.

[0037] Based on the globally optimal sorting order and the three-dimensional environment model, a collision-free motion trajectory for the collaborative robot is generated, and the collaborative robot is controlled to perform sorting operations according to the collision-free motion trajectory, including:

[0038] Based on the global optimal sorting order, the grasping position of each target object is determined, the alignment angle between the end effector of the collaborative robot and the grasping position is calculated, and the posture with the smallest alignment angle deviation is selected to obtain the joint angle configuration through inverse kinematics solution, thus forming the sorting motion trajectory.

[0039] The triangular mesh geometric models of other objects in the three-dimensional environment model, excluding the current target object, are marked as obstacle models. The directed distance from each vertex of the obstacle model to the reference origin is calculated. A spatial directed distance field is constructed by trilinear interpolation and the gradient direction is recorded.

[0040] The spatial trajectory point sequence is obtained by sampling the sorting motion trajectory. For each trajectory point, forward kinematics calculation is performed to obtain the Cartesian coordinates of the center point of each link. The distance value of the center point of each link in the spatial oriented distance field is queried. When the distance value is less than the safe distance threshold, it is marked as a collision risk point.

[0041] The gradient direction of the spatial oriented distance field is extracted from the collision risk point as the direction of the repulsive force. The magnitude of the repulsive force is calculated as the reciprocal of the difference between the safe distance threshold and the distance value. The product of the repulsive force direction and the magnitude of the repulsive force is applied to the joint angle configuration as a joint space correction vector. The correction is iterated until all distance values ​​are greater than the safe distance threshold. The corrected joint angle configuration sequence is converted into a collision-free motion trajectory and sent to the collaborative robot to perform the sorting operation.

[0042] A second aspect of this invention provides a deep learning-based collaborative robot object recognition and sorting system, comprising:

[0043] The image acquisition unit is used to acquire a continuous frame color image sequence of the area to be sorted and the corresponding three-dimensional spatial distance sequence.

[0044] The feature extraction unit is used to construct a spatiotemporal feature extraction network to perform temporal modeling on the continuous frame color image sequence, capture the motion trend features of the object through the inter-frame motion association module, and perform spatiotemporal joint encoding with the single-frame static appearance features to generate a dynamic feature representation containing motion prediction information.

[0045] A 3D modeling unit is used to perform point cloud reconstruction and registration processing on the 3D spatial distance sequence to generate a 3D environment model of the scene to be sorted. The 3D environment model includes object surface geometric information and spatial occupancy information.

[0046] The identification and determination unit is used to input the dynamic feature representation and the three-dimensional environment model into the fusion identification network to perform identity recognition and state determination on the object to be sorted, and obtain the object recognition result and motion state mark.

[0047] The task planning unit is used to establish a sorting dependency graph describing the spatial occlusion constraints and operation sequence constraints between objects through a collaborative task planner, based on the object recognition results and motion state markers, and to solve the globally optimal sorting order that satisfies the constraints based on the sorting dependency graph.

[0048] The trajectory generation unit is used to generate a collision-free motion trajectory for the collaborative robot based on the globally optimal sorting order and the three-dimensional environment model, and to control the collaborative robot to perform sorting operations according to the collision-free motion trajectory.

[0049] A third aspect of the present invention provides an electronic device, comprising:

[0050] processor;

[0051] Memory used to store processor-executable instructions;

[0052] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0053] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0054] This method significantly improves the object recognition and sorting capabilities of collaborative robots in complex dynamic scenarios. By acquiring continuous frame color image sequences and 3D spatial distance sequences, it achieves comprehensive perception of the sorting scene. The 3D spatial distance sequence, after point cloud reconstruction and registration, generates an accurate 3D environment model, fully preserving the surface geometric details and spatial occupancy information of objects. The dynamic feature representation and the 3D environment model work together in the fusion recognition network to not only identify objects but also accurately determine their current motion state.

[0055] Based on the recognition results and motion state markers, the collaborative task planner can intelligently analyze the spatial relationships between objects and construct a sorting dependency graph that includes spatial occlusion and operation sequence constraints. By solving this graph, the system can plan the globally optimal sorting sequence that satisfies all constraints. This process effectively solves the problems of repeated grabbing, path conflict, or task failure caused by improper operation sequence in traditional methods, ensuring the logical rationality and overall efficiency of sorting operations.

[0056] Ultimately, based on the globally optimal sorting sequence and a precise 3D environment model, the system plans a collision-free motion trajectory for the collaborative robot. This trajectory fully considers the geometric constraints between the robot body, the end effector, and the dynamic environment, ensuring the safe, smooth, and efficient execution of the sorting action. The entire method achieves closed-loop automation from perception, recognition, planning to execution, significantly improving the intelligence level, operational reliability, and overall throughput efficiency of the sorting system. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the collaborative robot object recognition and sorting method based on deep learning, as described in an embodiment of the present invention.

[0058] Figure 2 This is a flowchart illustrating a method for generating dynamic feature representations containing motion prediction information according to an embodiment of the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0061] Figure 1 This is a flowchart illustrating the deep learning-based collaborative robot object recognition and sorting method according to an embodiment of the present invention. Figure 1 As shown, the deep learning-based collaborative robot object recognition and sorting method includes:

[0062] Collect a continuous frame color image sequence of the area to be sorted and the corresponding three-dimensional spatial distance sequence;

[0063] A spatiotemporal feature extraction network is constructed to perform temporal modeling on the continuous frame color image sequence. The motion trend features of the object are captured by the inter-frame motion association module and spatiotemporally co-encoded with the static appearance features of a single frame to generate a dynamic feature representation containing motion prediction information.

[0064] The three-dimensional spatial distance sequence is reconstructed and registered using point cloud technology to generate a three-dimensional environment model of the scene to be sorted. The three-dimensional environment model includes geometric information of object surfaces and spatial occupancy information.

[0065] The dynamic feature representation and the three-dimensional environment model are input into the fusion recognition network to perform identity recognition and state determination on the objects to be sorted, and to obtain object recognition results and motion state markers.

[0066] Based on the object recognition results and motion state markers, a sorting dependency graph describing the spatial occlusion constraints and operation sequence constraints between objects is established through a collaborative task planner, and the globally optimal sorting sequence that satisfies the constraints is solved based on the sorting dependency graph.

[0067] Based on the globally optimal sorting order and the three-dimensional environment model, a collision-free motion trajectory for the collaborative robot is generated, and the collaborative robot is controlled to perform sorting operations according to the collision-free motion trajectory.

[0068] Figure 2This is a flowchart illustrating a method for generating dynamic feature representations containing motion prediction information according to an embodiment of the present invention. In one optional implementation, a spatiotemporal feature extraction network is constructed to perform temporal modeling on the continuous frame color image sequence. The motion trend features of objects are captured through an inter-frame motion correlation module and spatiotemporally co-encoded with single-frame static appearance features to generate a dynamic feature representation containing motion prediction information, including:

[0069] Pixel-level optical flow calculation is performed between adjacent frames in the continuous frame color image sequence. The displacement direction and displacement amplitude of each pixel are recorded to obtain a dense motion vector field. The spatial continuity analysis of the motion vector is used to identify the pixel set belonging to the same object and assign a motion entity identifier to each pixel set. The centroid trajectory curve of the pixel set corresponding to each motion entity identifier is extracted.

[0070] The centroid trajectory curve is fitted with a sliding window in the time dimension to obtain the rate of change of motion velocity and the deflection angle of motion direction, and the position distribution area of ​​each moving entity in the future is predicted to generate a motion trend feature vector carrying position prediction information.

[0071] For the current frame image in the continuous frame color image sequence, feature extraction is performed through a multi-layer convolutional neural network and a static appearance feature map is calculated. The static appearance feature map encodes the color texture and contour edge information of the object.

[0072] Based on the motion trend feature vector, the corresponding spatial region is located in the static appearance feature map, the feature response of the spatial region is amplified, and it is concatenated with the motion trend feature vector in the channel dimension and mapped to a dynamic feature representation of a unified dimension.

[0073] After acquiring a sequence of consecutive color images of the area to be sorted, in-depth spatiotemporal feature analysis is required to capture the dynamic behavior patterns of objects. Pixel-level optical flow calculation is performed between adjacent frames in the consecutive color image sequence. Optical flow calculation is a classic computer vision technique for detecting pixel motion in image sequences. Its basic principle is to assume that the grayscale values ​​of pixels remain unchanged between adjacent frames, and to solve for the motion vector of each pixel by establishing constraint equations for pixels in time and space. A deep learning-based optical flow estimation network is used. This network receives two frames of images at time t and time t+1 as input, and the output optical flow field F(x, y) represents the displacement vector of each pixel (x, y) in the image. This vector contains a horizontal displacement component u(x, y) and a vertical displacement component v(x, y). The optical flow field calculation process matches corresponding pixels in adjacent frames by constructing a cost volume. The cost volume is calculated at multiple scales to improve robustness to large displacement motions.

[0074] The dense motion vector field obtained through optical flow calculation contains motion information for each pixel in the image. However, the original pixel-level motion information is discrete and chaotic, requiring further analysis of the spatial continuity of the motion vectors to identify pixel sets belonging to the same object. Spatial clustering analysis is performed on the dense motion vector field. Specifically, the motion similarity between adjacent pixels is calculated, taking into account both the angle between their displacement directions and the difference in displacement amplitude. When the angle between the motion directions of two adjacent pixels is less than a preset angle threshold, and the difference in displacement amplitude is less than an amplitude threshold, these two pixels are considered to belong to the same moving entity. Using a region growing algorithm or graph cut method, pixels with similar motion are connected into connected regions. Each connected region represents an independent moving entity, and a unique motion entity identifier is assigned to each moving entity. This identifier remains consistent in subsequent temporal tracking to establish the object's motion trajectory in the time dimension.

[0075] After identifying each moving entity, the centroid trajectory curve of each entity needs to be extracted to characterize its overall motion pattern. For a set of pixels assigned a moving entity identifier, the centroid coordinates of this set in the image space are calculated. The centroid coordinates are obtained by weighted averaging of the coordinates of all pixels belonging to that entity, where the weights can be the optical flow amplitude or color gradient intensity of the pixels. In a series of consecutive frames, the centroid position of each moving entity in each frame is calculated sequentially. The centroid positions are then connected in chronological order to form the centroid trajectory curve. The centroid trajectory curve is a curve that evolves over time in the image space, and its shape characteristics reflect the motion pattern of the object. For example, a straight trajectory represents uniform motion, while a curved trajectory represents accelerated or turning motion.

[0076] To extract quantitative motion features from the centroid trajectory curve, a sliding window fitting is performed in the time dimension. A time window length is set, typically containing five to ten consecutive frames. Within this time window, a polynomial fit is performed on the centroid trajectory. The obtained polynomial curve can be used to calculate the velocity and acceleration. The velocity is calculated by taking the first-order time derivative of the fitted polynomial, representing the instantaneous velocity vector of the object at the current moment. The rate of change of velocity is obtained by taking the time derivative of the velocity vector, reflecting the magnitude of the object's acceleration. The deflection angle of the motion direction is obtained by calculating the angle between the velocity vectors at adjacent moments; this angle quantifies the trend of change in the object's motion direction.

[0077] Based on the fitted motion parameters, a kinematic model is used to predict the future position distribution regions of each moving entity. Assuming the object maintains its current rate of change of velocity and directional deflection angle over a short period, the centroid position for several future frames is extrapolated using the kinematic equations. Considering the uncertainty of the prediction, the predicted position is not a fixed point, but a probability distribution region, typically represented by a Gaussian distribution or an elliptical region. The major axis of the ellipse aligns with the predicted velocity direction, and its length is proportional to the velocity magnitude. The minor axis represents the lateral uncertainty of the prediction. The predicted position distribution region parameters are encoded into a vector form. This vector contains the predicted center coordinates, the geometric parameters of the distribution region, and dynamic information such as motion velocity and acceleration, forming a motion trend feature vector carrying position prediction information.

[0078] In parallel with dynamic motion information, it is necessary to extract static appearance features of objects in the image. For the current frame image in a continuous sequence of color images, feature extraction is performed using a multi-layer convolutional neural network. This convolutional neural network adopts a deep residual structure, containing a stack of multiple convolutional layers, batch normalization layers, and activation layers. Each convolutional operation uses convolutional kernels of different sizes to capture visual features at different scales. In the shallow layers of the network, smaller convolutional kernels and higher-resolution feature maps are used to capture detailed information such as the edges and textures of the object. In the deeper layers of the network, larger receptive fields and more abstract feature representations are used to encode the semantic information and overall contour of the object. The static appearance feature map is obtained through forward propagation. This feature map is a multi-channel three-dimensional tensor, whose spatial dimension corresponds to the spatial position of the input image, and the channel dimension encodes different types of visual features. The static appearance feature map encodes the object's color distribution features, texture patterns, and contour edge information. Static features are closely related to the object's category and appearance, but lack the ability to perceive the object's motion state.

[0079] To effectively fuse dynamic motion information with static appearance information, a spatial correspondence between the two needs to be established. Based on the object position information encoded in the motion trend feature vector, the corresponding spatial region is located in the static appearance feature map. Specifically, the position coordinates of the moving entity in image space are mapped to the spatial coordinate system of the feature map. Since convolutional neural networks typically include downsampling operations, the spatial resolution of the feature map is lower than that of the input image. Therefore, the coordinates need to be scaled and mapped according to the downsampling factor of the network. A local region centered on the mapped coordinates is extracted from the feature map. The size of this region is adaptively adjusted according to the scale of the moving entity. Attention weights are applied to the feature responses within this spatial region, with the attention weights related to the salience of motion at that location. Locations with higher motion speeds or significant acceleration changes are assigned higher attention weights. The feature responses of this region are amplified through element-wise multiplication. The amplification factor is dynamically adjusted according to the motion intensity in the motion trend feature vector, ensuring that regions with significant motion occupy a more important position in the feature representation.

[0080] The attention-weighted static appearance feature map contains spatial attention to moving objects, but still lacks explicit motion information encoding. Therefore, the motion trend feature vector is fused with the appearance features of the corresponding spatial region. This fusion operation is performed along the channel dimension. The motion trend feature vector is dimensionally expanded to match its spatial dimension with the local region of the appearance feature map. This is achieved through copying or interpolation. The motion trend feature vector and the appearance feature map are then concatenated along the channel dimension. The concatenated feature tensor contains more channels and encodes both static appearance information and dynamic motion information. Due to the high dimensionality of the concatenated feature, a one-by-one convolutional layer is used to compress the channel dimension, mapping the high-dimensional features to a unified-dimensional feature representation. This one-by-one convolutional layer learns the linear combination weights between different channels, automatically discovering the correlation between appearance features and motion features, and generating a compact fused feature representation.

[0081] The unified dimensional feature representation obtained by mapping is the dynamic feature representation. This representation maintains the same resolution as the feature map in the spatial dimension and has a fixed feature dimension in the channel dimension. The dynamic feature representation not only includes the static appearance information of the object at the current moment, but also integrates the object's motion history and future position prediction information over time. This spatiotemporal joint encoding feature representation can more accurately describe the state of objects in dynamic scenes, providing richer input information for subsequent object recognition and sorting decisions. Through end-to-end training, the parameters of the optical flow calculation module, motion segmentation module, trajectory prediction module, and feature fusion module in the spatiotemporal feature extraction network are jointly optimized, so that the generated dynamic feature representation is maximized to benefit downstream recognition and sorting tasks.

[0082] In one optional implementation, for the current frame image in the continuous frame color image sequence, feature extraction is performed using a multi-layer convolutional neural network to calculate a static appearance feature map. The static appearance feature map encodes the color texture and contour edge information of the object, including:

[0083] The current frame image is input into the first convolutional layer of a multi-layer convolutional neural network to perform preliminary feature extraction and obtain a shallow feature map. The shallow feature map is then separated into multiple independent feature channel response maps, each of which corresponds to a specific color mode or texture direction.

[0084] The shallow feature map is passed to the second and third convolutional layers for progressive feature abstraction. The texture composite pattern is captured by expanding the receptive field in the second convolutional layer, and the shape contour features and geometric structure features of the object are extracted in the third convolutional layer to obtain the deep feature map.

[0085] The gradient magnitude of the deep feature map is calculated to obtain the edge intensity response distribution. The edge intensity response distribution identifies the feature activation intensity of the object contour position. By setting an edge saliency threshold, edge feature regions with high activation intensity are filtered out to generate a contour edge enhancement mask.

[0086] The contour edge enhancement mask is applied to the shallow feature map to enhance the ability of color and texture information to be expressed at the edge position. At the same time, the contour edge enhancement mask is fused with the deep feature map through element-wise multiplication to highlight the contour edge features. The enhanced shallow feature map and the fused deep feature map are then spliced ​​together in the channel dimension to generate a static appearance feature map.

[0087] In this specific embodiment, the extraction of static appearance features from the continuous frame color image sequence forms the basis for subsequent spatiotemporal modeling. For the input current frame image, its resolution is 640×480 pixels, and the pixel values ​​are normalized to the [0, 1] interval, forming a three-channel RGB color image tensor.

[0088] The color image tensor is input into a multi-layer convolutional neural network architecture. The first convolutional layer uses 64 convolutional kernels with a kernel size of 7×7, a stride of 2, and a 3-pixel boundary padding mode to ensure reasonable downsampling of spatial dimensions. This convolutional layer is followed by a batch normalization layer and a ReLU activation function. The output shallow feature map has a size of 320×240×64. The shallow feature map mainly responds to the low-level visual features of the image, including local color changes, simple texture directions, and preliminary contour cues.

[0089] The shallow feature maps were processed by channel separation, decomposing the 64 feature channels into independent response maps. Each feature channel response map maintained a spatial resolution of 320×240 but contained only the feature activation values ​​of a single channel. Visual analysis revealed that some feature channels were highly responsive to red areas, some were sensitive to vertical textures, and some generated strong activations to circular contour structures. This channel separation mechanism provides room for subsequent selective feature enhancement, allowing for targeted enhancement of specific visual patterns.

[0090] The shallow feature maps are then passed to the second convolutional layer, which contains 128 kernels. The kernel size is reduced to 3×3, with a stride of 1 and padding of 1. The output feature map size is 320×240×128. The receptive field of this layer is expanded to 15×15 pixels compared to the first layer, enabling it to capture texture composite patterns over a larger spatial range. For example, for woven surfaces, the second convolutional layer can simultaneously encode the interlacing structure of warp and weft threads; for metallic surfaces, this layer can integrate reflective textures from multiple directions to form a comprehensive representation. The second convolutional layer is also configured with batch normalization and ReLU activation to enhance the network's training stability and non-linear expressive power.

[0091] The third convolutional layer further enhances the level of feature abstraction. This layer uses 256 convolutional kernels, maintaining a kernel size of 3×3, with a stride of 2 for spatial downsampling and padding of 1. The output deep feature map size is 160×120×256, expanding the network's effective receptive field to 31×31 pixels, sufficient to cover the complete outline of a medium-sized object. The deep feature map encodes the shape and outline features of the object, including geometric structural features such as rectangles, circles, and polygons. It also responds to complex geometric attributes such as the object's uneven surfaces and boundary corners. Each spatial location in the deep feature map actually corresponds to a 4×4 pixel region in the original image, forming a hierarchical feature representation from local details to overall structure.

[0092] Gradient magnitudes are calculated for deep feature maps using the Sobel operator to compute the gradient responses in the horizontal and vertical directions. The horizontal Sobel kernel is [-1, 0, 1; -2, 0, 2; -1, 0, 1], and the vertical Sobel kernel is [-1, -2, -1; 0, 0, 0; 1, 2, 1]. Sobel convolutions are applied to each of the 256 channels of the deep feature map to obtain horizontal and vertical gradient tensors, both with dimensions of 160×120×256. The magnitudes of the two gradient tensors are calculated channel-wise. At each spatial location (i, j) and channel c, the gradient magnitude is... ,in This represents the horizontal gradient response value. This represents the vertical gradient response value.

[0093] The resulting gradient magnitude tensor represents the edge intensity response distribution. Its value directly indicates the feature activation intensity at the object's contour location. In the object's boundary region, the gradient magnitude increases significantly due to drastic changes in feature values; in flat areas inside the object or in the background region, the gradient magnitude approaches zero. To filter out true edge feature regions, an edge salience threshold is set. Thresholding is applied to the gradient magnitude tensor when... When the target location is high activation intensity, the location is marked as a high activation intensity edge feature point and the mask value is set to 1; otherwise, the mask value is set to 0, generating a contour edge enhancement mask with a size of 160×120×256. This mask clearly identifies the spatial location and channel distribution of the object contour in binary form.

[0094] To enhance the expressive power of the shallow feature map at edge locations, the contour edge enhancement mask needs to be aligned with the shallow feature map. Since the shallow feature map has a spatial size of 320×240 and the deep feature map has a size of 160×120, there is a two-fold resolution difference. Therefore, bilinear interpolation is used to upsample the contour edge enhancement mask, restoring it to a size of 320×240×256. Because the shallow feature map has 64 channels and the mask has 256 channels, a channel-dimensional broadcast copy is performed, applying the mask information of the first 64 channels to the shallow feature map. At each spatial location, the feature value of the shallow feature map is multiplied by the corresponding mask value. When the mask value is 1, the original feature value is preserved and enhanced; when the mask value is 0, the feature response at that location is weakened. This element-wise multiplication operation selectively enhances color and texture information at object edges, highlighting the color contrast and texture details in the edge regions.

[0095] Meanwhile, the unsampled contour edge enhancement mask is fused with the deep feature map through element-wise multiplication. The deep feature map has a size of 160×120×256, which perfectly matches the mask size. Element-wise multiplication is performed directly. The fused deep feature map maintains high feature activation values ​​at the contour edges, while feature responses are suppressed in non-edge regions. This operation highlights shape contour features and geometric structure features, making object boundary information the dominant component of feature representation.

[0096] The enhanced shallow feature map and the fused deep feature map are integrated into a unified static appearance feature map. Due to the inconsistent spatial dimensions of the two, the enhanced shallow feature map is downsampled by a factor of 2, and max pooling is performed with a 2×2 kernel size and a stride of 2, adjusting the size to 160×120×64. At this point, the size of the downsampled shallow feature map is 160×120×64, and the size of the fused deep feature map is 160×120×256. A concatenation operation is performed along the channel dimension, connecting the two feature maps along the third dimension to generate a static appearance feature map with a size of 160×120×320. The first 64 channels of this feature map encode color and texture information enhanced by edges, and the last 256 channels encode shape and geometry information highlighted by contours, forming a complete representation from low-level visual features to high-level semantic features.

[0097] The static appearance feature map comprehensively encodes the multi-level visual attributes of an object. Color and texture information supports the discrimination of the object's material category, such as distinguishing between plastic and metal parts. Contour edge information provides precise positioning of the object's shape and provides object boundary constraints in the two-dimensional image domain for subsequent fusion with three-dimensional spatial information. As a single-frame static feature representation, this feature map will be spatiotemporally co-encoded with the motion trend features extracted by the inter-frame motion association module to form a dynamic feature representation containing motion prediction information, laying the foundation for object identification and state determination in the fusion recognition network.

[0098] In one optional implementation, point cloud reconstruction and registration processing is performed on the three-dimensional spatial distance sequence to generate a three-dimensional environment model of the scene to be sorted. The three-dimensional environment model includes object surface geometric information and spatial occupancy information, including:

[0099] The three-dimensional spatial distance sequence is recombined into a structured depth image according to the scanning time sequence and angular resolution of the sensor. Bilateral filtering is performed to smooth the distance measurement noise and maintain the depth jump characteristics of the object edge. The filtered depth image is then back-projected into three-dimensional space to generate an ordered point cloud.

[0100] The ordered point cloud is subjected to point-by-point normal vector estimation. The surface normal vector direction of each point is calculated by using the ordered structure of the point cloud through the local tangent plane formed by adjacent rows and columns. The geometric discontinuity boundary of the object surface is detected according to the rate of change of the angle between the normal vector directions of adjacent points. The ordered point cloud is divided into multiple connected region point cloud segments along the geometric discontinuity boundary.

[0101] For each connected region point cloud segment, calculate its spatial envelope boundary. The minimum convex polyhedron is obtained by calculating the convex hull of the boundary points. The volume of the minimum convex polyhedron is marked as the spatial volume of the corresponding object. The shape similarity matching of connected region point cloud segments of adjacent acquisition frames is performed to establish the inter-frame correspondence.

[0102] The rigid body transformation parameters between point cloud fragments in connected regions of adjacent frames are calculated based on the inter-frame correspondence. The point cloud fragments in connected regions of multiple frames are aligned and fused to form a complete point cloud representation. The point cloud fragments are then reconstructed into a mesh to generate a triangular mesh geometric model of the object surface. The spatial volume is then combined to generate a three-dimensional environment model.

[0103] After acquiring the three-dimensional spatial distance sequence of the area to be sorted, the distance measurement data needs to be converted into a three-dimensional geometric representation that can be used for scene understanding. The three-dimensional spatial distance sequence is usually acquired by depth sensors such as LiDAR, structured light cameras, or time-of-flight cameras. The sensors periodically measure the distance values ​​in each direction in the scene according to a fixed scanning pattern. The original distance measurement data is arranged according to the scanning sequence of the sensors. Based on the horizontal and vertical angular resolution of the sensors, the one-dimensional distance data stream is reconstructed into a structured depth image in the form of a two-dimensional matrix. During the reconstruction process, the mapping relationship between each distance measurement value and its corresponding scanning angle is maintained, so that each pixel position in the depth image corresponds to a specific direction in the sensor coordinate system.

[0104] Due to sensor measurement noise and environmental interference, random measurement errors exist in the original depth image. Bilateral filtering is applied to the depth image, performing weighted smoothing in both the spatial and numerical domains. The spatial domain weights are determined based on the Euclidean distance between pixels, while the numerical domain weights are determined based on the depth value differences. Bilateral filtering can smooth depth noise while preserving the depth transition characteristics at object edges, avoiding edge blurring caused by traditional Gaussian filtering. For each pixel in the depth image, a fixed-size neighborhood window is selected centered on that pixel. The spatial and numerical domain weights of each pixel within the window relative to the center pixel are calculated. These two weights are multiplied to obtain a comprehensive weight. The filtered depth value is then obtained by weighted averaging of the depth values ​​of the neighboring pixels.

[0105] The filtered depth image contains smoothed distance measurements, which need to be converted into point coordinates in three-dimensional space. Based on the sensor's intrinsic calibration parameters and the position of each pixel in the depth image, the corresponding three-dimensional coordinates are calculated using the inverse projection relationship of the pinhole camera model. Specifically, for a pixel at position (u, v) in the depth image, its depth value is d, and the sensor focal length is f. x and f y The optical center is located at (c x c y If the pixel's coordinates are 1, then the corresponding 3D coordinates are 1. , , Perform the above calculation on all valid pixels in the depth image to generate a set of three-dimensional points. Since the three-dimensional points maintain the row and column organization structure of the depth image, it is called an ordered point cloud.

[0106] After obtaining the ordered point cloud, it is necessary to extract the geometric properties of the object's surface, estimate the normal vector for each point in the point cloud, and simplify the calculation process by utilizing the neighborhood structure characteristics of the ordered point cloud. It has explicit row indexes in the point cloud data structure. and column indexes It can directly obtain its four adjacent points. , , , Using two points in adjacent rows and Constructing vectors Using two points in adjacent columns and Constructing vectors The cross product of these two vectors Given the direction of the surface normal vector at that point, normalize the normal vector to obtain the unit normal vector.

[0107] Based on normal vector information, geometric discontinuities on object surfaces can be detected. At object edges, the boundary between an object and a supporting plane, and the contact points between different objects, the direction of the surface normal vector changes significantly. The angle between the normal vector of each point in the point cloud and the normal vectors of its neighboring points is calculated, and the cosine of the angle is obtained by calculating the dot product of the two unit normal vectors. When the angle between the normal vectors of adjacent points exceeds a set threshold, a geometric discontinuity boundary is determined to exist at that location. The ordered point cloud's row and column structure is traversed, and all detected boundary points are marked. These boundary points divide the point cloud into multiple connected regions. Connectivity analysis is performed on the point cloud. Starting from any unmarked point, region growing is performed, grouping all points connected by adjacency relationships and not crossing boundaries into the same connected region. This process is repeated until all points are assigned to a connected region, resulting in multiple connected region point cloud fragments. Each fragment corresponds to an object surface or an independent geometric part in the scene.

[0108] For each connected region point cloud segment, its spatial occupancy is calculated. Boundary points of this segment are extracted; these are points located at the edge of the segment and not adjacent to other regions. A convex hull calculation is performed on the set of boundary points to find the minimum convex polyhedron containing all boundary points. An incremental construction algorithm is used for the convex hull calculation. Four non-coplanar boundary points are selected to form an initial tetrahedron, and the remaining boundary points are added sequentially. For each newly added point, it is determined whether it is inside the current convex hull; if it is outside, the facets of the convex hull are updated. After the convex hull calculation is complete, the convex polyhedron encloses all surface points of the corresponding object, and its volume approximately represents the spatial volume of the object. When calculating the volume of the convex polyhedron, the polyhedron is decomposed into multiple tetrahedrons sharing a common vertex. The volumes of each tetrahedron are calculated separately and then summed. The calculated spatial volume is marked as the spatial occupancy information of the object corresponding to the connected region point cloud segment.

[0109] Since the sensor acquires a series of three-dimensional spatial distance sequences across multiple consecutive frames, it is necessary to establish the correlation between corresponding objects in different frames. This involves matching connected point cloud segments in adjacent frames, using shape feature similarity to determine whether point cloud segments in different frames correspond to the same object, and extracting the shape descriptor for each point cloud segment, including its centroid position, principal direction, and aspect ratio. The shape similarity between point cloud segments in adjacent frames is calculated, taking into account centroid distance, principal direction angle, and size ratio differences. For each point cloud segment in the current frame, the point cloud segment with the highest similarity exceeding the matching threshold in the previous frame is searched to establish the inter-frame correspondence.

[0110] After establishing the inter-frame correspondence, the rigid body transformation parameters between corresponding point cloud segments are calculated. Assuming that the same object only undergoes rigid body motion between adjacent frames, which can be described by 3D rotation and translation, an iterative nearest-point registration algorithm is used to calculate the rigid body transformation parameters. This algorithm alternately executes two steps: corresponding point pair search and transformation parameter optimization. Under the current transformation estimation, for each point in the source point cloud, the nearest point in the target point cloud is found to form a corresponding point pair. Based on minimizing the sum of squared distances between all corresponding point pairs, the rotation matrix R and translation vector t are solved. This process is iteratively executed until the transformation parameters converge or the maximum number of iterations is reached. The calculated rotation matrix and translation vector describe the motion transformation of the object from the previous frame to the current frame.

[0111] Using the calculated rigid body transformation parameters, point cloud fragments from multiple connected regions are aligned to a unified coordinate system. A reference frame is selected as the global coordinate system, and point cloud fragments from other frames are mapped to the reference coordinate system through corresponding rigid body transformations. For point cloud fragments with inter-frame correspondences, they are fused after transformation alignment, merging point cloud data from different frames that correspond to the same object to form a complete point cloud representation of the object. During the point cloud fusion process, points in overlapping areas are deduplicated, retaining points with higher measurement quality among those spatially close points, and removing redundant points to reduce the amount of data.

[0112] A complete point cloud represents a densely sampled point on the object's surface. Further construction of the object's surface geometry is needed. This involves triangulating the complete point cloud to generate a surface mesh model composed of triangular facets. The algorithm employs either the Moving Cube algorithm or the Poisson surface reconstruction algorithm. The former constructs the mesh based on the intersection of the implicit surface and a regular mesh, while the latter fits the gradient field of the indicator function of the point cloud by solving the Poisson equation. The Poisson surface reconstruction algorithm first constructs a vector field based on the point cloud and its normal vectors. This vector field points outwards from the object's surface and inwards from the surface. Solving the Poisson equation yields a scalar indicator function, which takes a positive value inside the object and a negative value outside. The zero isosurface represents the object's surface. The triangular mesh corresponding to the zero isosurface is extracted; this triangular mesh geometric model describes the continuous geometry of the object's surface.

[0113] By combining the spatial volume information corresponding to point cloud fragments with the triangular mesh geometric model, a complete 3D environment model is constructed. Each identified object in the 3D environment model is described by its surface geometry and spatial occupancy information. The surface geometry information, provided by the triangular mesh model, includes the 3D coordinates of each point on the object's surface and the surface normal vector, used for subsequent grasping position planning and collision detection. The spatial occupancy information is described by the object's spatial volume and convex hull boundary, representing the object's occupied range in 3D space, used to determine spatial occupancy relationships between objects and accessibility analysis of the robot's trajectory. The 3D environment model integrates the geometric representation and spatial relationships of all objects to be sorted in the scene.

[0114] In one optional implementation, the dynamic feature representation and the three-dimensional environment model are input into a fusion recognition network to perform identity recognition and state determination on the objects to be sorted, obtaining object recognition results and motion state markers, including:

[0115] The dynamic feature is extracted as a vector difference between consecutive time steps, which is used as an inter-frame motion increment and accumulated by time window to obtain motion trajectory features. The object position in the three-dimensional environment model is matched and associated with the timestamp of the motion trajectory features and converted into spatial position features.

[0116] Calculate the correlation coefficient matrix between the motion trajectory features and the spatial location features, and perform singular value decomposition to extract the left singular vector and right singular vector corresponding to the principal singular value. Using the left singular vector and right singular vector as the basis, linear projection is performed on the motion trajectory features and spatial location features to obtain motion pattern projection coefficients and spatial pattern projection coefficients. The two sets of projection coefficients are multiplied element-wise to generate coupled features.

[0117] The coupling features are input into the identity recognition channel and the state determination channel respectively. The identity recognition channel extracts the spatial distribution characteristic components, which are then mapped to the category label space after global average pooling to obtain the object recognition result. The state determination channel extracts the temporal evolution characteristic components, which are then mapped to the state category space after temporal convolution to obtain the motion state label.

[0118] In this specific embodiment, considering the temporal characteristics of dynamic feature representation, a difference operation is performed on the feature vectors of consecutive time steps to extract inter-frame change information. Specifically, between time steps t and t+1, the dynamic feature vector f is calculated. t with f t+1 The difference Δf t =f t+1 -f t This difference vector represents the motion increment in the feature space. To fully capture the motion trend of the object, a fixed-length time window is set, typically 8 to 16 frames in size. Within this time window, the motion increments between all frames are accumulated to form the motion trajectory feature M = [Δf1, Δf2, ..., Δf...]. N ], where N is the number of frames within the time window. This accumulation method can effectively filter out instantaneous noise interference and retain stable motion pattern information.

[0119] Simultaneously, spatial position information of objects is extracted from the 3D environment model. This model includes geometric attributes such as the object's center coordinates, bounding box vertex coordinates, and surface normals, obtained through point cloud reconstruction. For each detected object in the scene, its 3D centroid coordinates (x, y, z) are extracted. i y i , z i The spatial occupancy of an object is calculated by combining the point cloud density distribution. To establish the correspondence between motion features and spatial features, timestamp information needs to be matched with spatial location. Specifically, the timestamp of each time step in the motion trajectory features is synchronized with the acquisition time of the 3D environment model, and temporal interpolation is used to ensure the temporal consistency of the features. For the timestamp-aligned spatial location sequence P=[(x1, y1, z1), (x2, y2, z2), ... , (x N y N , z NThe three-dimensional coordinates are encoded into a spatial location feature vector. The encoding process can adopt a multilayer perceptron structure to map the three-dimensional coordinates to a feature space with the same dimension as the motion trajectory features, forming a spatial location feature matrix S.

[0120] Establishing the correlation between motion trajectory features and spatial location features is a key step in achieving effective fusion. This involves calculating the correlation coefficient matrix between the two. The elements of the matrix Indicating motion characteristics Dimension and spatial features The degree of correlation between dimensions. To extract the dominant pattern in the correlation matrix, the degree of correlation between dimensions is determined. Perform singular value decomposition to obtain ,in and These are the left singular matrix and the right singular matrix, respectively. It is a diagonal singular value matrix. Before selection... The singular vectors corresponding to the largest singular values ​​are usually The value is taken as 60% to 80% of the rank of the correlation matrix. Extract the left singular matrix. The former The column constitutes the base of the motion pattern Extracting the right singular matrix The former Columns constitute the spatial pattern base .

[0121] The original features are projected using the basis matrix obtained from singular value decomposition. This transforms the motion trajectory features... Projected onto the motion pattern substrate, the motion pattern projection coefficients are calculated. This projection coefficient reflects the distribution of motion features across the dominant motion mode. Similarly, spatial location features... Project onto the spatial pattern substrate and calculate the spatial pattern projection coefficients. These projection coefficients characterize the response strength of spatial features to the dominant spatial pattern. Both sets of projection coefficients have the same dimension. To achieve deep feature fusion, [the following is a separate, unrelated sentence:] ... and Perform element-wise multiplication, i.e., Hadamard product operation. This product operation can capture the interactive response between motion patterns and spatial patterns, generating feature representations that contain motion-space coupling information.

[0122] Coupling characteristics This system contains both object identification information and motion state description information, but the representation methods of these two types of information differ significantly. To extract different types of information separately, a dual-channel parallel processing architecture is designed. The identification channel focuses on extracting the spatial distribution characteristics of the object, which are mainly reflected in the object's position, shape, and texture in the scene. Through multi-layer convolution operations, spatial distribution characteristic components are extracted from the coupled features. The convolution kernel size is set to 3×3, and the number of channels increases layer by layer, forming a feature abstraction process from shallow to deep layers. After multi-layer convolution, a feature map containing rich spatial semantics is obtained. Global average pooling is performed on this feature map to aggregate the spatial dimension information, resulting in a fixed-length global feature vector. A fully connected layer maps the global feature vector to the category label space. The dimension of the category label space is equal to the number of predefined object categories. The softmax activation function is used to normalize the output vector to obtain the probability distribution of each category. The category with the highest probability value is selected as the object recognition result.

[0123] The state determination channel focuses on extracting the temporal evolution characteristics of an object, which reflect the motion and change patterns of the object over continuous time. Separating the temporal evolution component from the coupled features requires preserving the temporal dimension information of the features. A temporal convolutional network is used to model the temporal features. Temporal convolution ensures that the prediction at the current moment depends only on historical information through causal convolution. A multi-layer temporal convolutional structure is set, with each layer using dilated convolution to expand the receptive field, enabling the network to capture motion patterns over a long period. The dilation rate increases exponentially, with the dilation rate of the l-th layer set to 2. l After temporal convolution, temporal features encoding the motion evolution law are obtained. These temporal features are then mapped to a state category space, which contains discrete state labels such as stationary, uniform motion, accelerated motion, and decelerated motion. The softmax function is also used for probability normalization, and the confidence of each state is output. The state with the highest confidence is selected as the motion state label.

[0124] The dual-channel network employs a multi-task learning strategy during training, simultaneously optimizing both the identity recognition loss and the state determination loss. The identity recognition loss uses a cross-entropy loss function to measure the difference between the predicted class distribution and the true label. The state determination loss also uses cross-entropy, but class weights are introduced to adjust for the imbalanced distribution of motion states. The total loss function is a weighted sum of the losses from the two sub-tasks, with the weight coefficients adjusted according to the importance and difficulty of the tasks. During training, the network parameters are updated through backpropagation, enabling the fusion recognition network to simultaneously learn the ability to discriminate between object identities and motion states.

[0125] To enhance the robustness of recognition, an attention mechanism is introduced during feature fusion. The importance weights of each dimension of the coupled features are calculated, key feature dimensions are strengthened, and redundant or noisy dimensions are suppressed. The attention weights are generated through a lightweight gating network. This network takes the coupled features as input and outputs a weight vector with the same feature dimensions. The weight vector is multiplied element-wise with the coupled features to achieve adaptive feature enhancement.

[0126] Through the above-mentioned fusion recognition process, the temporal motion information in the dynamic feature representation and the spatial geometric information in the three-dimensional environment model can be fully utilized to achieve accurate identification of the objects to be sorted and reliable determination of their motion status. The object recognition results provide object category information for subsequent sorting task planning, while the motion status marking provides real-time motion prediction for safe grasping in dynamic scenarios. Together, they constitute the perception foundation of the collaborative robot intelligent sorting system.

[0127] In one optional implementation, based on the object recognition results and motion state markers, a sorting dependency graph describing spatial occlusion constraints and operation sequence constraints between objects is established using a cooperative task planner. The globally optimal sorting order satisfying the constraints is then solved based on the sorting dependency graph, including:

[0128] Based on the three-dimensional environment model, extract the triangular mesh geometric model of each object, emit grasping path rays from the current position of the end effector of the collaborative robot to the centroid of each object, calculate the number of intersecting faces of each ray with the triangular mesh geometric model of other objects, and establish occlusion constraint edges when the number of intersecting faces exceeds the occlusion determination threshold. Connect the occluded object and the occluding object in the dependency graph and label the occlusion constraint type.

[0129] For objects identified as being in motion, their historical position sequences are extracted, and acceleration and velocity coefficients are obtained by fitting a quadratic function using the least squares method. The probability density function of position prediction is calculated, and the variance of the probability density function is used as the position uncertainty. Based on the position uncertainty, an operation urgency value is assigned to the object, and operation order constraint edges are established in the dependency graph with the operation urgency value attached.

[0130] A depth-first traversal is performed on the sorting dependency graph. During the traversal, a stack of visited nodes is maintained. When an edge pointing to an existing node in the stack is encountered, a loop structure is identified. The operation urgency values ​​of all constraint edges in the loop are extracted. The constraint edge with the smallest operation urgency value is deleted. The traversal is re-executed until no loop exists. The final node access sequence is taken as the globally optimal sorting order.

[0131] After obtaining the recognition results and motion state markers of the objects to be sorted, a sorting dependency graph needs to be established through a collaborative task planner to determine a reasonable grasping order. The collaborative task planner first performs spatial occlusion analysis on all objects in the sorting scene. Specifically, the triangular mesh geometric model of each object is extracted from the 3D environment model. This triangular mesh geometric model consists of a set of vertex coordinates, a list of face indices, and normal vector information, which completely describes the surface geometry of the object. When extracting the triangular mesh geometric model, the moving cube algorithm is used to mesh the point cloud data, and the mesh resolution is set to 2 mm to ensure that the geometric model can accurately reflect the true contour of the object.

[0132] The occlusion relationship is determined based on the ray casting method. The current position of the collaborative robot's end effector is used as the ray origin, which is calculated through the robot's forward kinematics and includes the actuator's three-dimensional coordinates in the world coordinate system. For each object in the sorting area, its centroid coordinates need to be calculated. The centroid coordinates are obtained by taking the arithmetic mean of the coordinates of all points in the object's point cloud. A grasping path ray is emitted from the actuator position toward the object's centroid. This ray is represented in three-dimensional space as a parametric equation, and the ray direction vector is obtained by normalizing the difference between the target centroid coordinates and the origin coordinates.

[0133] During ray propagation, it is necessary to detect whether the ray intersects with the triangular mesh geometry of all other objects in the scene. The intersection detection uses the Möller-Trumbore algorithm, which determines whether the ray passes through a specific triangular mesh by solving the parametric equations of the ray and the triangular mesh. For each grabbing path ray, the number of intersecting surfaces with the triangular mesh geometry of other objects is counted. When the ray passes through an object, the incident point and the exit point correspond to two intersecting surfaces. Therefore, completely penetrating an object usually produces two or more intersecting surfaces.

[0134] The occlusion threshold setting needs to consider the physical size and safety margin of the end effector. In practical applications, the occlusion threshold is usually set to 3, meaning that when the number of intersecting faces exceeds 3, the target object is considered to be occluded by other objects. In this case, an occlusion constraint edge is created in the dependency graph. This constraint edge points from the occluding object node to the occluded object node, indicating that the occluding object must be removed before the occluded object can be captured. The occlusion constraint edge is labeled with a constraint type, marked "Spatial Occlusion Constraint." This label is used to distinguish different types of dependencies during subsequent constraint conflict handling.

[0135] For objects identified as moving in the recognition results, it is necessary to establish operation sequence constraints based on motion prediction and extract the historical position sequence of the moving objects. This sequence consists of the centroid coordinates of the object in the most recent 8 consecutive frames, with the time interval being the reciprocal of the camera's frame rate. The historical position sequence forms a motion trajectory in 3D space. Motion modeling is performed on each coordinate component of this trajectory, and a quadratic function is fitted using the least squares method. The quadratic function has the form p(t) = at 2 +vt+p0, where p(t) represents the position coordinate, t represents time, a is the acceleration coefficient, v is the velocity coefficient, and p0 is the initial position. The fitting process involves constructing an overdetermined system of equations and using the normal equation method to solve for the optimal parameters.

[0136] After obtaining the parameters of the quadratic function, the probability density function for position prediction can be derived. Assuming the fitting residuals follow a zero-mean Gaussian distribution, the predicted value of the object's position at future time steps is $p$. pred (t future )=at future 2 +vt future +p0, the prediction uncertainty is obtained by calculating the variance of the fitted residuals, and the location uncertainty is denoted as... Its value reflects the predictability of an object's trajectory. Objects with stable motion patterns have small fitting residuals and low positional uncertainty; while objects with complex trajectories or sudden changes in direction have high positional uncertainty.

[0137] Operational urgency is assigned to moving objects based on their positional uncertainty. The calculation of operational urgency considers both positional uncertainty and the object's velocity. The velocity magnitude is obtained using the Euclidean norm of the velocity coefficient v. Objects with high positional uncertainty or high velocity should be prioritized for grabbing to prevent them from moving into hard-to-reach areas. When calculating the operational urgency value, both the positional uncertainty and velocity magnitude are normalized to the range of 0 to 1. The two are then weighted and summed to obtain the final operational urgency value. The weighting coefficients are adjusted according to the actual application scenario; typically, the positional uncertainty weight is set to 0.6, and the velocity weight is set to 0.4.

[0138] When establishing operation order constraint edges in the dependency graph, objects with higher operation urgency values ​​are set as priority grabbing targets. Specifically, all moving objects are arranged in descending order of operation urgency values. Directed edges are established in the dependency graph from object nodes with high urgency to object nodes with low urgency. These directed edges are marked as "operation order constraints" and the operation urgency value is attached as the weight attribute of the edge. The existence of operation order constraint edges indicates that in the sorting sequence, the object corresponding to the starting node should be placed before the object corresponding to the ending node.

[0139] After constructing the complete sorting dependency graph, it is necessary to check whether there is a loop structure in the graph. The existence of a loop means that there is a conflict between constraints, and it is impossible to satisfy all the constraints. A depth-first traversal algorithm is used to perform topological sorting on the dependency graph. During the traversal, a stack of visited nodes is maintained. This stack records all nodes on the current traversal path. Starting from any unvisited node in the graph, the node is pushed onto the stack and marked as visited. Then, the adjacent nodes are traversed along the outgoing edges of the node. When a node is traversed, it is checked whether the node already exists in the stack of visited nodes. If it does, it means that a loop structure has been found.

[0140] The strategy for handling loop structures is to remove the weakest constraint edge in the loop, thereby breaking the circular dependency. All constraint edges in the loop are then extracted. For occlusion constraint edges, their constraint strength is determined by the number of intersecting faces between the occluding and occluded objects; a higher number of intersecting faces indicates a more severe occlusion and a stronger constraint. For operation order constraint edges, their constraint strength is directly represented by the additional operation urgency value. The constraint strength values ​​of all constraint edges in the loop are normalized, and then the edge with the weakest constraint strength is selected for deletion. After deletion, a depth-first traversal is performed again to check for other loops. This process is iterated until the dependency graph becomes a directed acyclic graph (DAG).

[0141] After obtaining the directed acyclic graph, a globally optimal sorting order is generated using a topological sorting algorithm. Topological sorting is based on the post-order traversal of a depth-first search, starting with nodes with an in-degree of zero. Each traversed node is added sequentially to the sorted sequence. The final node access sequence represents the sorting order that satisfies all constraints. Objects at the beginning of this sequence are prioritized for grabbing, while those at the end are processed later. This globally optimal sorting order, while ensuring occlusion relationships are satisfied, prioritizes objects with high motion uncertainty, effectively reducing the risk of grabbing failures due to object movement. It also avoids unnecessary waiting time and improves the overall efficiency of the sorting operation.

[0142] In one optional implementation, based on the globally optimal sorting order and the three-dimensional environment model, a collision-free motion trajectory for the collaborative robot is generated, and the collaborative robot is controlled to perform sorting operations according to the collision-free motion trajectory, including:

[0143] Based on the global optimal sorting order, the grasping position of each target object is determined, the alignment angle between the end effector of the collaborative robot and the grasping position is calculated, and the posture with the smallest alignment angle deviation is selected to obtain the joint angle configuration through inverse kinematics solution, thus forming the sorting motion trajectory.

[0144] The triangular mesh geometric models of other objects in the three-dimensional environment model, excluding the current target object, are marked as obstacle models. The directed distance from each vertex of the obstacle model to the reference origin is calculated. A spatial directed distance field is constructed by trilinear interpolation and the gradient direction is recorded.

[0145] The spatial trajectory point sequence is obtained by sampling the sorting motion trajectory. For each trajectory point, forward kinematics calculation is performed to obtain the Cartesian coordinates of the center point of each link. The distance value of the center point of each link in the spatial oriented distance field is queried. When the distance value is less than the safe distance threshold, it is marked as a collision risk point.

[0146] The gradient direction of the spatial oriented distance field is extracted from the collision risk point as the direction of the repulsive force. The magnitude of the repulsive force is calculated as the reciprocal of the difference between the safe distance threshold and the distance value. The product of the repulsive force direction and the magnitude of the repulsive force is applied to the joint angle configuration as a joint space correction vector. The correction is iterated until all distance values ​​are greater than the safe distance threshold. The corrected joint angle configuration sequence is converted into a collision-free motion trajectory and sent to the collaborative robot to perform the sorting operation.

[0147] After obtaining the globally optimal sorting order, the discrete task sequence needs to be transformed into continuous motion control commands executable by the collaborative robot. For each target object to be grasped, the object surface point cloud is extracted from the 3D environment model, the principal direction axis of the object is calculated through principal component analysis, and a stable grasping surface is determined by combining prior knowledge of the object category. The determination of the grasping position needs to comprehensively consider the object's center of gravity position, surface curvature distribution, and geometric constraints of the end effector. Candidate grasping points are sampled on the object surface and the grasping quality index of each candidate point is evaluated. The point with the highest quality score is selected as the target grasping position. The grasping quality index comprehensively considers the contact area, force sealing, and grasping stability to ensure that the actuator can reliably grip the object.

[0148] After determining the grasping position, it is necessary to calculate the optimal pose of the end effector when it reaches that position. The pose of the end effector is usually represented by a rotation matrix or quaternion, describing the spatial orientation of the actuator coordinate system relative to the base coordinate system. The ideal pose should make the gripping direction of the actuator coincide with the normal vector of the object surface, while avoiding the joints from exceeding the range of motion limits. Since there are usually multiple solutions for inverse kinematics, we enumerate all feasible solutions and calculate the angular deviation between the actuator pose and the ideal grasping direction for each solution. The joint angle configuration with the smallest deviation is selected as the target pose. The inverse kinematics solution uses analytical or numerical iteration methods to map the target pose of the end effector to the specific values ​​of each joint angle, forming a discrete path point sequence of the robot from the current position to the grasping position and then to the placement position.

[0149] To achieve obstacle avoidance path planning, all objects in the 3D environment model that are not the currently operated object are considered obstacles. Each obstacle is represented by a triangular mesh model, containing a sequence of vertex coordinates and topological connections. A fixed reference point in the scene is selected as the origin. The Euclidean distance from each vertex on the obstacle surface to this origin is calculated, and the distance value is assigned a positive or negative sign depending on whether the vertex is inside or outside the obstacle. Points outside the obstacle have positive distances, and points inside have negative distances. By dividing the entire workspace into regular 3D meshes, a directed distance field value is calculated at each mesh node using trilinear interpolation. This allows any location in the space to obtain the signed distance to the nearest obstacle surface by querying the values ​​of neighboring mesh nodes and interpolating. Simultaneously, the 3D gradient vector of the directed distance field at each mesh node is recorded, with the gradient direction pointing in the direction of the fastest increase in distance value, i.e., the direction away from the obstacle surface.

[0150] Collision detection is performed on the initial sorting trajectory obtained from the planning. A dense sequence of trajectory points is generated on the joint space trajectory using an equal-interval sampling strategy. The sampling interval is dynamically adjusted according to the robot's motion speed and environmental complexity. For each sampled joint angle configuration, forward kinematics transformation is performed to calculate the spatial pose of each link of the robot. The forward kinematics calculation is based on the cascade multiplication of link parameters and rotation transformation matrices to obtain the three-dimensional Cartesian coordinates of the center points of all intermediate links from the base to the end effector. Each link is simplified into a cylinder or enveloping sphere model, and the geometric center of the link is taken as the representative point for collision detection. The value of this point in the pre-constructed spatial directed distance field is queried.

[0151] Collision risk assessment is achieved by comparing the directed distance field query value with a preset safe distance threshold. The safe distance threshold is usually set as the link radius plus an additional safety margin to ensure that the robot always maintains a physical distance from obstacles. When the directed distance field value of the center point of a link is less than the threshold, it indicates that the distance between the link and the obstacle is insufficient and there is a potential collision risk. The trajectory point is marked as a collision risk point and trajectory correction is required. After performing traversal detection on all trajectory points, the number and spatial distribution of collision risk points are counted to provide a basis for subsequent optimization.

[0152] For detected collision risk points, a gradient-based trajectory optimization process is initiated. The gradient vector of the spatially directed distance field at the collision risk point is extracted as the direction of the repulsive force, indicating the fastest path away from the obstacle. The magnitude of the repulsive force is designed to be the reciprocal of the difference between the safe distance threshold and the actual distance, so that trajectory points closer to the obstacle experience a stronger repulsive force, forming a nonlinear force field response characteristic. The repulsive force direction unit vector is multiplied by the repulsive force magnitude scalar to obtain the repulsive force vector in Cartesian space. The repulsive force in Cartesian space is mapped to the joint space using the pseudo-inverse of the robot's Jacobian matrix, generating correction increments for each joint angle. This correction vector is proportional to the repulsive force and inversely proportional to the joint velocity influence coefficient, ensuring smooth correction actions without causing abrupt joint changes.

[0153] The calculated joint space correction vector is superimposed on the original joint angle configuration to update the joint parameters of the trajectory points. After correction, forward kinematics calculation and collision detection are re-executed. It is checked whether the directed distance field values ​​of all link center points are greater than the safe distance threshold. If collision risk points still exist, the gradient extraction, repulsion force calculation, and joint correction process is repeated to form an iterative optimization loop. An adaptive step size mechanism is introduced during the iteration process to adjust the amplitude of the correction vector according to the changing trend of the number of collision risk points in continuous iterations, which accelerates convergence and prevents oscillation. When the directed distance field values ​​of all sampled trajectory points meet the safety constraints or the preset maximum number of iterations is reached, the iteration process is terminated, and the corrected joint angle configuration sequence is output as the final collision-free motion trajectory.

[0154] To ensure the executability of the trajectory, the optimized joint angle sequence is smoothed. A cubic spline interpolation method is used to generate continuous joint angle curves between adjacent trajectory points, ensuring that the joint angular velocity and angular acceleration are continuously differentiable throughout the motion process, avoiding mechanical impact caused by sharp turns. At the same time, it is checked whether the interpolated joint velocity and acceleration exceed the kinematic limits of each joint of the robot. The parts exceeding the limits are stretched on a time scale. By extending the motion time, the peak velocity and acceleration are reduced to meet the hardware constraints.

[0155] The processed collision-free motion trajectory is converted into a joint control command format recognizable by the robot controller. These commands include the target angle, target angular velocity, and target torque information for each joint at each time stamp. A position-velocity hybrid control mode is employed to achieve precise trajectory tracking. The control command sequence is sent to the collaborative robot's underlying controller via a real-time communication interface. The controller then drives the joint motors to perform corresponding actions based on the commands. During execution, the actual position feedback and torque sensor signals of each joint are continuously monitored. If an unexpected collision is detected or the joint error exceeds the allowable range, an emergency stop mechanism is immediately triggered to ensure operational safety. After the grasping and placement of a single object is completed, the position state of the object in the 3D environment model is updated, and the geometric information of the sorted objects is removed. This provides an accurate environmental representation for trajectory planning of the next target object. All tasks in the globally optimal sorting sequence are executed sequentially until the entire batch of objects is sorted.

[0156] A second aspect of this invention provides a deep learning-based collaborative robot object recognition and sorting system, comprising:

[0157] The image acquisition unit is used to acquire a continuous frame color image sequence of the area to be sorted and the corresponding three-dimensional spatial distance sequence.

[0158] The feature extraction unit is used to construct a spatiotemporal feature extraction network to perform temporal modeling on the continuous frame color image sequence, capture the motion trend features of the object through the inter-frame motion association module, and perform spatiotemporal joint encoding with the single-frame static appearance features to generate a dynamic feature representation containing motion prediction information.

[0159] A 3D modeling unit is used to perform point cloud reconstruction and registration processing on the 3D spatial distance sequence to generate a 3D environment model of the scene to be sorted. The 3D environment model includes object surface geometric information and spatial occupancy information.

[0160] The identification and determination unit is used to input the dynamic feature representation and the three-dimensional environment model into the fusion identification network to perform identity recognition and state determination on the object to be sorted, and obtain the object recognition result and motion state mark.

[0161] The task planning unit is used to establish a sorting dependency graph describing the spatial occlusion constraints and operation sequence constraints between objects through a collaborative task planner, based on the object recognition results and motion state markers, and to solve the globally optimal sorting order that satisfies the constraints based on the sorting dependency graph.

[0162] The trajectory generation unit is used to generate a collision-free motion trajectory for the collaborative robot based on the globally optimal sorting order and the three-dimensional environment model, and to control the collaborative robot to perform sorting operations according to the collision-free motion trajectory.

[0163] A third aspect of the present invention provides an electronic device, comprising:

[0164] processor;

[0165] Memory used to store processor-executable instructions;

[0166] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0167] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0168] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based collaborative robot object recognition sorting method, characterized in that, include: Collect a continuous frame color image sequence of the area to be sorted and the corresponding three-dimensional spatial distance sequence; A spatiotemporal feature extraction network is constructed to perform temporal modeling on the continuous frame color image sequence. The motion trend features of the object are captured by the inter-frame motion association module and spatiotemporally co-encoded with the static appearance features of a single frame to generate a dynamic feature representation containing motion prediction information. The three-dimensional spatial distance sequence is reconstructed and registered using point cloud technology to generate a three-dimensional environment model of the scene to be sorted. The three-dimensional environment model includes geometric information of object surfaces and spatial occupancy information. The dynamic feature representation and the three-dimensional environment model are input into the fusion recognition network to perform identity recognition and state determination on the objects to be sorted, and to obtain object recognition results and motion state markers. Based on the object recognition results and motion state markers, a sorting dependency graph describing the spatial occlusion constraints and operation sequence constraints between objects is established through a collaborative task planner, and the globally optimal sorting sequence that satisfies the constraints is solved based on the sorting dependency graph. Based on the globally optimal sorting order and the three-dimensional environment model, a collision-free motion trajectory for the collaborative robot is generated, and the collaborative robot is controlled to perform sorting operations according to the collision-free motion trajectory.

2. The method of claim 1, wherein, A spatiotemporal feature extraction network is constructed to perform temporal modeling on the continuous frame color image sequence. The motion trend features of objects are captured through an inter-frame motion correlation module and spatiotemporally co-encoded with single-frame static appearance features to generate a dynamic feature representation containing motion prediction information, including: Pixel-level optical flow calculation is performed between adjacent frames in the continuous frame color image sequence. The displacement direction and displacement amplitude of each pixel are recorded to obtain a dense motion vector field. The spatial continuity analysis of the motion vector is used to identify the pixel set belonging to the same object and assign a motion entity identifier to each pixel set. The centroid trajectory curve of the pixel set corresponding to each motion entity identifier is extracted. The centroid trajectory curve is fitted with a sliding window in the time dimension to obtain the rate of change of motion velocity and the deflection angle of motion direction, and the position distribution area of ​​each moving entity in the future is predicted to generate a motion trend feature vector carrying position prediction information. For the current frame image in the continuous frame color image sequence, feature extraction is performed through a multi-layer convolutional neural network and a static appearance feature map is calculated. The static appearance feature map encodes the color texture and contour edge information of the object. Based on the motion trend feature vector, the corresponding spatial region is located in the static appearance feature map, the feature response of the spatial region is amplified, and it is concatenated with the motion trend feature vector in the channel dimension and mapped to a dynamic feature representation of a unified dimension.

3. The method of claim 2, wherein, For the current frame image in the continuous frame color image sequence, feature extraction and static appearance feature map are performed using a multi-layer convolutional neural network. The static appearance feature map encodes the color texture and contour edge information of the object, including: The current frame image is input into the first convolutional layer of a multi-layer convolutional neural network to perform preliminary feature extraction and obtain a shallow feature map. The shallow feature map is then separated into multiple independent feature channel response maps, each of which corresponds to a specific color mode or texture direction. The shallow feature map is passed to the second and third convolutional layers for progressive feature abstraction. The texture composite pattern is captured by expanding the receptive field in the second convolutional layer, and the shape contour features and geometric structure features of the object are extracted in the third convolutional layer to obtain the deep feature map. The gradient magnitude of the deep feature map is calculated to obtain the edge intensity response distribution. The edge intensity response distribution identifies the feature activation intensity of the object contour position. By setting an edge saliency threshold, edge feature regions with high activation intensity are filtered out to generate a contour edge enhancement mask. The contour edge enhancement mask is applied to the shallow feature map to enhance the ability of color and texture information to be expressed at the edge position. At the same time, the contour edge enhancement mask is fused with the deep feature map through element-wise multiplication to highlight the contour edge features. The enhanced shallow feature map and the fused deep feature map are then spliced ​​together in the channel dimension to generate a static appearance feature map.

4. The method of claim 1, wherein, The three-dimensional spatial distance sequence is reconstructed and registered using point clouds to generate a three-dimensional environment model of the scene to be sorted. This three-dimensional environment model includes object surface geometry information and spatial occupancy information, including: The three-dimensional spatial distance sequence is recombined into a structured depth image according to the scanning time sequence and angular resolution of the sensor. Bilateral filtering is performed to smooth the distance measurement noise and maintain the depth jump characteristics of the object edge. The filtered depth image is then back-projected into three-dimensional space to generate an ordered point cloud. The ordered point cloud is subjected to point-by-point normal vector estimation. The surface normal vector direction of each point is calculated by using the ordered structure of the point cloud through the local tangent plane formed by adjacent rows and columns. The geometric discontinuity boundary of the object surface is detected according to the rate of change of the angle between the normal vector directions of adjacent points. The ordered point cloud is divided into multiple connected region point cloud segments along the geometric discontinuity boundary. For each connected region point cloud segment, calculate its spatial envelope boundary. The minimum convex polyhedron is obtained by calculating the convex hull of the boundary points. The volume of the minimum convex polyhedron is marked as the spatial volume of the corresponding object. The shape similarity matching of connected region point cloud segments of adjacent acquisition frames is performed to establish the inter-frame correspondence. The rigid body transformation parameters between point cloud fragments in connected regions of adjacent frames are calculated based on the inter-frame correspondence. The point cloud fragments in connected regions of multiple frames are aligned and fused to form a complete point cloud representation. The point cloud fragments are then reconstructed into a mesh to generate a triangular mesh geometric model of the object surface. The spatial volume is then combined to generate a three-dimensional environment model.

5. The method of claim 1, wherein, The dynamic feature representation and the three-dimensional environment model are input into a fusion recognition network to perform identity recognition and state determination on the objects to be sorted, obtaining object recognition results and motion state markers, including: The dynamic feature is extracted as a vector difference between consecutive time steps, which is used as an inter-frame motion increment and accumulated by time window to obtain motion trajectory features. The object position in the three-dimensional environment model is matched and associated with the timestamp of the motion trajectory features and converted into spatial position features. Calculate the correlation coefficient matrix between the motion trajectory features and the spatial location features, and perform singular value decomposition to extract the left singular vector and right singular vector corresponding to the principal singular value. Using the left singular vector and right singular vector as the basis, linear projection is performed on the motion trajectory features and spatial location features to obtain motion pattern projection coefficients and spatial pattern projection coefficients. The two sets of projection coefficients are multiplied element-wise to generate coupled features. The coupling features are input into the identity recognition channel and the state determination channel respectively. The identity recognition channel extracts the spatial distribution characteristic components, which are then mapped to the category label space after global average pooling to obtain the object recognition result. The state determination channel extracts the temporal evolution characteristic components, which are then mapped to the state category space after temporal convolution to obtain the motion state label.

6. The method of claim 1, wherein, Based on the object recognition results and motion state markers, a sorting dependency graph describing spatial occlusion constraints and operation sequence constraints between objects is established using a cooperative task planner. The globally optimal sorting order satisfying the constraints is then solved based on this sorting dependency graph, including: Based on the three-dimensional environment model, extract the triangular mesh geometric model of each object, emit grasping path rays from the current position of the end effector of the collaborative robot to the centroid of each object, calculate the number of intersecting faces of each ray with the triangular mesh geometric model of other objects, and establish occlusion constraint edges when the number of intersecting faces exceeds the occlusion determination threshold. Connect the occluded object and the occluding object in the dependency graph and label the occlusion constraint type. For objects identified as being in motion, their historical position sequences are extracted, and acceleration and velocity coefficients are obtained by fitting a quadratic function using the least squares method. The probability density function of position prediction is calculated, and the variance of the probability density function is used as the position uncertainty. Based on the position uncertainty, an operation urgency value is assigned to the object, and operation order constraint edges are established in the dependency graph with the operation urgency value attached. A depth-first traversal is performed on the sorting dependency graph. During the traversal, a stack of visited nodes is maintained. When an edge pointing to an existing node in the stack is encountered, a loop structure is identified. The operation urgency values ​​of all constraint edges in the loop are extracted. The constraint edge with the smallest operation urgency value is deleted. The traversal is re-executed until no loop exists. The final node access sequence is taken as the globally optimal sorting order.

7. The method of claim 1, wherein, Based on the globally optimal sorting order and the three-dimensional environment model, a collision-free motion trajectory for the collaborative robot is generated, and the collaborative robot is controlled to perform sorting operations according to the collision-free motion trajectory, including: Based on the global optimal sorting order, the grasping position of each target object is determined, the alignment angle between the end effector of the collaborative robot and the grasping position is calculated, and the posture with the smallest alignment angle deviation is selected to obtain the joint angle configuration through inverse kinematics solution, thus forming the sorting motion trajectory. The triangular mesh geometric models of other objects in the three-dimensional environment model, excluding the current target object, are marked as obstacle models. The directed distance from each vertex of the obstacle model to the reference origin is calculated. A spatial directed distance field is constructed by trilinear interpolation and the gradient direction is recorded. The spatial trajectory point sequence is obtained by sampling the sorting motion trajectory. For each trajectory point, forward kinematics calculation is performed to obtain the Cartesian coordinates of the center point of each link. The distance value of the center point of each link in the spatial oriented distance field is queried. When the distance value is less than the safe distance threshold, it is marked as a collision risk point. The gradient direction of the spatial oriented distance field is extracted from the collision risk point as the direction of the repulsive force. The magnitude of the repulsive force is calculated as the reciprocal of the difference between the safe distance threshold and the distance value. The product of the repulsive force direction and the magnitude of the repulsive force is applied to the joint angle configuration as a joint space correction vector. The correction is iterated until all distance values ​​are greater than the safe distance threshold. The corrected joint angle configuration sequence is converted into a collision-free motion trajectory and sent to the collaborative robot to perform the sorting operation.

8. A deep learning based collaborative robotic object recognition sorting system for implementing the method of any one of claims 1-7, characterized by, include: The image acquisition unit is used to acquire a continuous frame color image sequence of the area to be sorted and the corresponding three-dimensional spatial distance sequence. The feature extraction unit is used to construct a spatiotemporal feature extraction network to perform temporal modeling on the continuous frame color image sequence, capture the motion trend features of the object through the inter-frame motion association module, and perform spatiotemporal joint encoding with the single-frame static appearance features to generate a dynamic feature representation containing motion prediction information. A 3D modeling unit is used to perform point cloud reconstruction and registration processing on the 3D spatial distance sequence to generate a 3D environment model of the scene to be sorted. The 3D environment model includes object surface geometric information and spatial occupancy information. The identification and determination unit is used to input the dynamic feature representation and the three-dimensional environment model into the fusion identification network to perform identity recognition and state determination on the object to be sorted, and obtain the object recognition result and motion state mark. The task planning unit is used to establish a sorting dependency graph describing the spatial occlusion constraints and operation sequence constraints between objects through a collaborative task planner, based on the object recognition results and motion state markers, and to solve the globally optimal sorting sequence that satisfies the constraints based on the sorting dependency graph. The trajectory generation unit is used to generate a collision-free motion trajectory for the collaborative robot based on the globally optimal sorting order and the three-dimensional environment model, and to control the collaborative robot to perform sorting operations according to the collision-free motion trajectory.

9. An electronic device, comprising: include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon computer program instructions, wherein, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images