Parcel association method and device, electronic device, and storage medium

Abstract

Embodiments of the present application provide a parcel association method and device, electronic equipment and storage medium, belonging to the technical field of logistics sorting. The method comprises: acquiring a to-be-detected image, and determining a to-be-associated parcel in the to-be-detected image; determining a candidate associated parcel corresponding to the to-be-associated parcel; determining a motion feature similarity according to a position deviation between an actual detection position and a predicted position of the to-be-associated parcel; performing a tumbling detection on the to-be-associated parcel, and if the to-be-associated parcel tumbles, acquiring a motion speed of the to-be-associated parcel; if the motion speed of the to-be-associated parcel is less than a threshold, performing feature comparison on an appearance feature of the to-be-associated parcel and a feature library of the candidate associated parcel, and determining an appearance feature similarity; fusing the motion feature similarity and the appearance feature similarity to obtain a first association score; and if the first association score is not less than the threshold, associating the to-be-associated parcel with the candidate associated parcel. The embodiments of the present application can improve the accuracy of tumbling parcel association.

Description

Technical Field

[0001] This application relates to the field of logistics sorting technology, and in particular to a parcel association method and apparatus, electronic equipment and storage medium. Background Technology

[0002] In automated logistics sorting systems, parcels are scanned by mainline scanning equipment to collect label information before falling into the corresponding sorting chute. To achieve fully automated and intelligent management, the parcels must maintain an accurate association with the label information after entering the chute. This allows backend personnel or automated equipment to identify the identity information of stacked parcels without repeated scanning, effectively improving sorting efficiency and reducing logistics error rates caused by misidentification or mispacking.

[0003] Currently, related technologies typically employ the deployment of multiple 2D or 3D vision cameras along the chute to achieve end-to-end identity binding of packages from the main line and chute to the final cage loading stage through real-time multi-view tracking. This process mainly includes two core components: continuous tracking within a single camera's field of view and identity relay across camera fields of view. In the single-camera continuous tracking phase, packages often tumble, causing changes in the observation surface and fundamentally altering their appearance. If matching is still based on appearance features in such scenarios, tracking errors or identity loss are likely to occur. Summary of the Invention

[0004] The main objective of this application is to provide a package association method, apparatus, electronic device, and storage medium that can improve the accuracy of package association during tumbling.

[0005] To achieve the above objectives, a first aspect of this application proposes a package association method, which is applied to package tracking within a single camera. The method includes: Acquire the image to be detected captured by the camera, and perform package detection on the image to be detected to determine at least one package to be associated in the image to be detected; For each package to be associated, candidate associated packages corresponding to the package to be associated are determined from the tracked packages contained in the previous frame image captured by the camera corresponding to the image to be detected. The predicted position is obtained by predicting the position of the candidate associated wrapper in the image to be detected, and the motion feature similarity is determined based on the positional deviation between the actual detected position of the candidate associated wrapper in the image to be detected and the predicted position. Perform a roll detection on the package to be associated. If the roll detection result indicates that the package to be associated has rolled, obtain the movement speed of the package to be associated in the image to be detected. If the movement speed of the package to be associated is less than a preset threshold, the appearance features of the package to be associated in the image to be detected are compared with the feature library of the candidate associated packages to determine the appearance feature similarity. The feature library includes the appearance features of the candidate associated packages under multiple preset viewpoints. The motion feature similarity and the appearance feature similarity are fused to obtain the first association score between the package to be associated and the candidate associated package; If the first association score is not less than a preset threshold, the package to be associated is associated with the candidate associated package. To achieve the above objective, a second aspect of this application provides a package association device applied to package tracking within a single camera, the device comprising: The package detection unit is used to acquire the image to be detected captured by the camera, and to perform package detection on the image to be detected to determine at least one package to be associated in the image to be detected. The candidate association determination unit is used to determine, for each package to be associated, a candidate associated package corresponding to the package to be associated from the tracked packages contained in the previous frame image corresponding to the image to be detected, captured by the camera. The position deviation determination unit is used to predict the position of the candidate associated package in the image to be detected, and to determine the motion feature similarity based on the position deviation between the actual detected position and the predicted position of the package in the image to be detected. The roll detection unit is used to perform roll detection on the package to be associated. If the roll detection result indicates that the package to be associated has rolled, the unit obtains the speed of the package to be associated in the image to be detected. The appearance comparison unit is used to compare the appearance features of the package to be associated in the image to be detected with the feature library of candidate associated packages if the movement speed of the package to be associated is less than a preset threshold, and to determine the appearance feature similarity. The feature library includes the appearance features of candidate associated packages under multiple preset viewpoints. The association score calculation unit is used to fuse motion feature similarity and appearance feature similarity to obtain the first association score between the package to be associated and the candidate associated packages; In the package association unit, if the first association score is not less than a preset threshold, the package to be associated is associated with the candidate associated package.

[0006] To achieve the above objectives, a third aspect of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in any one of the embodiments of the first aspect.

[0007] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of the embodiments of the first aspect.

[0008] The package association method, apparatus, electronic device, and storage medium proposed in this application significantly improve the accuracy of package association in tumbling scenarios through a collaborative mechanism of roll detection triggering, speed-sensing re-identification timing control, and multi-view feature library matching. First, the introduced roll detection can promptly trigger the re-identification process when a fundamental change in appearance occurs. It dynamically decides based on the movement speed, performing appearance matching only when the speed is below a preset threshold and the image is clear. This effectively avoids unreliable matching caused by image blurring and point cloud degradation during high-speed movement, fundamentally reducing the risk of false association. Second, the feature library used contains the appearance features of candidate packages under multiple preset views. This ensures that regardless of which side the package is facing after rollover, a corresponding view template can be found in the feature library for comparison, overcoming the inherent problem of traditional methods where appearance features cannot be matched due to changes in the observation surface. Attached Figure Description

[0009] Figure 1 This is a flowchart of one embodiment of the package association method provided in this application; Figure 2 This is a flowchart of an embodiment of rollover detection provided in this application; Figure 3 This is a flowchart of an embodiment of the feature library construction method provided in this application; Figure 4 yes Figure 1 A flowchart of an embodiment for determining the similarity of appearance features in step S105; Figure 5 This is a flowchart of an embodiment of the feature library expansion method provided in this application; Figure 6 yes Figure 1 Flowchart of another embodiment for determining the similarity of appearance features in step S105; Figure 7 This is a schematic diagram of the package association device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0011] In automated logistics sorting systems, parcels are scanned by mainline scanning equipment to collect label information before falling into the corresponding sorting chute. To achieve fully automated and intelligent management, the parcels must maintain an accurate association with the label information after entering the chute. This allows backend personnel or automated equipment to identify the identity information of stacked parcels without repeated scanning, effectively improving sorting efficiency and reducing logistics error rates caused by misidentification or mispacking.

[0012] Currently, related technologies typically employ the deployment of multiple 2D or 3D vision cameras along the chute to achieve end-to-end identity binding of packages from the main line and chute to the final cage loading stage through real-time multi-view tracking. This process mainly includes two core components: continuous tracking within a single camera's field of view and identity relay across camera fields of view. In the single-camera continuous tracking stage, package tumbling is one of the main causes of tracking breakdowns in logistics sorting chutes, posing the following technical challenges: 1. A fundamental mutation in appearance: After a package is tumbled, the surfaces observed by the camera change completely. Taking a standard six-sided package as an example: Before tumbling: front (with label); After tumbling: side or back (without label or different sides). This results in a severe mismatch between RGB features and point cloud features and historical features observed during tracking.

[0013] 2. Confusing tumbling with loss: The system faces the difficulty of distinguishing between the following two scenarios: Scenario A: The package is tampered with, and the ID should remain unchanged; Scenario B: The original package is lost, and a new package appears, and the ID should change.

[0014] 3. Viewpoint alignment issues in 3D images: The main 3D image covers five sides of data (excluding the bottom side), but it is unknown which side the chute camera is currently observing.

[0015] 4. Motion blur during the tumbling process: When a package is tumbling at high speed, motion blur occurs in the imaging: RGB images become blurred, ReID feature extraction becomes unreliable; point cloud data becomes sparse or deformed, resulting in inaccurate geometric features; and appearance feature matching is prone to failure during rapid movement.

[0016] 5. Partially visible transitional states: During the tumbling process, the package may be in a transitional posture, with only part of its surface visible, resulting in incomplete feature extraction.

[0017] Based on this, embodiments of this application provide a package association method and apparatus, electronic device and storage medium, which can improve the accuracy of associated tumbling packages.

[0018] The package association method provided in this application relates to the field of logistics sorting technology. The package association method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, barcode scanner, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application implementing the package association method, but is not limited to the above forms.

[0019] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network personal computers (PCs), minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0020] Please see Figure 1 , Figure 1 This is an optional flowchart of the package association method provided in the embodiments of this application. In some embodiments of this application, the package association method is applied to package tracking within a single camera. Figure 1 The specific methods may include, but are not limited to, steps S101 to S107.

[0021] Step S101: Obtain the image to be detected captured by the camera, and perform package detection on the image to be detected to determine at least one package to be associated in the image to be detected. Step S102: For each package to be associated, determine the candidate associated packages corresponding to the package to be associated from the tracked packages contained in the previous frame image corresponding to the image to be detected captured by the camera. Step S103: Predict the position of the candidate associated package in the image to be detected to obtain the predicted position, and determine the motion feature similarity based on the positional deviation between the actual detected position and the predicted position of the package in the image to be detected. Step S104: Perform roll detection on the package to be associated. If the roll detection result indicates that the package to be associated has rolled, obtain the motion speed of the package to be associated in the image to be detected. Step S105: If the movement speed of the package to be associated is less than a preset threshold, the appearance features of the package to be associated in the image to be detected are compared with the feature library of the candidate associated packages to determine the appearance feature similarity. The feature library includes the appearance features of the candidate associated packages under multiple preset viewpoints. Step S106: The motion feature similarity and appearance feature similarity are fused to obtain the first association score between the package to be associated and the candidate associated packages; Step S107: If the first association score is not less than a preset threshold, associate the package to be associated with the candidate associated package.

[0022] In step S101 of some embodiments, the end-to-end identity binding of the package from the main line, chute to the final cage loading stage is realized. This mainly includes two core aspects: continuous tracking within a single camera's field of view and identity relay across camera fields of view. Continuous tracking within a single camera's field of view refers to maintaining the stable continuation of the package's tracking identity (Track ID) through cross-frame target matching within a continuous frame sequence of the same camera, ensuring that the binding relationship between the package and the waybill information does not break when the package moves within the coverage area of ​​a single camera. Identity relay across camera fields of view refers to transferring the package's identity identifier from the previous camera to the next camera when the package moves along the chute, leaves the current camera's field of view, and enters the detection area of ​​the next adjacent camera. This is achieved through target re-identification or coordinate system calibration and trajectory prediction between different cameras, realizing seamless identity inheritance during camera switching. It should be noted that the method provided in this application embodiment is mainly used to solve the problem of tracking errors or identity loss when the appearance and other features of the package become unstable due to package tumbling in a continuous tracking scenario within a single camera's field of view. The method provided in this application embodiment can run independently in any single 3D camera deployed along the sorting chute, serving as the core matching engine of the camera's internal tracking module. When a package enters the chute from the main sorting line via the swing wheel or chute sorting mechanism and first appears within the detection field of view of the first camera along the chute, the camera identifies and registers the package using a target detection model, assigns it an initial tracking identity, and binds this identity to the waybill number reported by the main line six-sided scanning system, thereby triggering continuous tracking of the package. Subsequently, throughout the entire process of the package moving along the chute, each time it enters the coverage area of ​​a subsequent camera, the consistency of its cross-frame identity can be maintained within that camera based on the association strategy defined in the method provided in this application embodiment, until the package moves to the end of the chute and enters the cage-loading or temporary storage stage. For ease of explanation, the following description uses a camera deployed at any position along the chute as an example.

[0023] Specifically, firstly, the image to be detected, captured by a 3D camera, can be read in real time. This image can contain two-dimensional RGB color information, or corresponding depth information or three-dimensional point cloud data. Then, a deep learning-based object detection model (such as the YOLO series) can be used to perform forward inference computation (i.e., package detection) on the image. By regressing bounding boxes and class confidence, the detection region of each individual package is segmented and determined from the image background. Newly appearing packages in the image that have not yet established an identity with the previous frame are defined as packages to be associated. For example, when the image to be detected is the current frame (t), a package to be associated could refer to a newly observed target in the current frame that needs to be matched with a known tracked package from the previous frame (t-1) to continue or establish an identity.

[0024] In step S102 of some embodiments, the historical state information of all tracked packages that have been successfully assigned a unique tracking identifier (Track ID) in the previous frame image corresponding to the image to be detected can first be retrieved from memory or cache. This historical state information may at least include the detection box position coordinates, velocity estimates, and corresponding feature vector caches of the package in the previous frame image. To avoid the O(n²) computational cost caused by exhaustively pairing each package to be associated with the image to be detected with all tracked packages in the previous frame image, this embodiment introduces a tracking algorithm for preliminary association and candidate screening. Specifically, the tracking algorithm may employ a simple threshold filter based on a motion model or a lightweight online tracker (e.g., the Kalman filter prediction stage in Sort or DeepSort): for each tracked package in the previous frame image, its predicted position in the current frame (i.e., the frame corresponding to the image to be detected) is derived based on its state vector and motion model in the previous frame, and the Mahalanobis distance or intersection-over-union (IoU) overlap between the predicted position and the detection positions of each package to be associated in the current frame is calculated. Then, packets falling within the prediction threshold (i.e., Mahalanobis distance less than a certain lenient threshold, or IoU greater than a preset low threshold) are considered candidate matches for the tracked packets in the current frame. In other words, the tracked packets are used as candidate associated packets for the packet to be associated, thus establishing a preliminary mapping relationship from the packet to be associated to the candidate associated packets. For example, if the previous frame contains three tracked packets (A, B, and C), and the image to be detected contains three packets to be associated (D, E, and F), the tracking algorithm can initially determine that packets A are associated with packet D, B with packet E, and C with packet F. That is, packet A is considered a candidate associated packet of packet D, packet B with packet E, and packet C with packet F. It is understood that although the preliminary association results determined by the tracking algorithm may have deviations, these results are only used to establish preliminary correspondences. Subsequent steps will further verify and correct these associations through fine-grained feature fusion. This method of preliminary association based on tracking algorithms effectively removes a large number of spatially distant and obviously impossible-to-pair invalid combinations by introducing motion continuity priors, reducing the computational scale of subsequent fine feature comparison to the constant level or linear matching range, thereby significantly reducing the real-time processing pressure of the system.

[0025] In step S103 of some embodiments, for each package to be associated (e.g., package D), its kinematic similarity to a candidate associated package (e.g., package A) is calculated. Specifically, the system can reuse and run a Kalman filter bound to the candidate associated packages, which maintains a Kalman filter containing the two-dimensional coordinates (x, y) of the target center and its velocity components. , The system uses the state transition matrix of the filter to predict the position of the candidate associated object in the current frame of the image to be detected based on the motion information of the candidate object wrapped in historical frames. The prediction result is the predicted position. At the same time, the center coordinates of the detection box output by the target detection model in step S101 can be directly read and used as the actual detection position to be associated in the current frame. Then, the positional deviation between the actual detection position and the predicted position can be calculated, and the motion feature similarity can be determined based on the positional deviation. In a feasible implementation, the motion feature similarity can be measured by Mahalanobis distance. Mahalanobis distance can take into account the uncertainty covariance matrix generated by Kalman filter prediction, thereby adaptively adjusting the distance tolerance under different confidence levels. Specifically, Mahalanobis distance can be mapped to the [0,1] interval through an exponential function or Gaussian kernel function. The smaller the distance, the closer the motion similarity is to 1, indicating that the motion trajectory is continuous and highly consistent.

[0026] In step S104 of some embodiments, after completing motion feature comparison, a rollover detection can be performed on the package to be associated to determine whether the package has undergone a significant change in posture, i.e., whether it has rolled over. Rollover can refer to the phenomenon where a package rotates around its own center while moving in a chute, causing a fundamental change in the surface of the package observed by the same camera. Rollover detection can be based on joint determination of multi-dimensional signals, such as appearance feature abrupt change detection, positional continuity detection, and morphological change detection. All detection signals are calculated based on continuous frame data from the same camera. If the rollover detection result indicates that the package to be associated has rolled over, the motion speed of the package in the image to be detected is further obtained. Motion speed (denoted as...) The value can be obtained by calculating the displacement of the actual detection position between adjacent frames (i.e., the position deviation) and dividing it by the frame interval time (Δt), or it can be extracted directly from the velocity component of the Kalman filter.

[0027] Reference Figure 2 In some embodiments, the "roll detection of the associated package" in step S104 may include, but is not limited to, steps S201 to S203.

[0028] Step S201: Determine the appearance change feature value based on the appearance features of the package to be associated in the image to be detected and the appearance features of the candidate associated package in the previous frame image. Step S202: Based on the image to be detected and the previous frame image, determine the point cloud shape change or detection box shape change of the package to be associated. Step S203: If the position deviation is less than a preset threshold, the appearance change feature value is less than a preset threshold, and the point cloud shape change or detection box shape change is greater than a preset threshold, generate a tumbling detection result indicating that the package to be associated has tumbled.

[0029] In step S201 of some embodiments, the detection of abrupt changes in appearance features is described. First, the appearance feature vector to be associated in the current frame of the image to be detected is obtained, as well as the appearance feature vector of the corresponding candidate associated wrapper in historical frames (e.g., the previous frame image or the average of several recent frames). The appearance feature vector can be a deep embedding feature extracted by inputting the cropped and normalized wrapped RGB image into a pre-trained ReID network (such as OsNet, MGN, ResNet, etc.). Its dimension is usually 256 or 512, and it is normalized by the L2 norm. Then, the cosine similarity between the two appearance feature vectors can be calculated. The similarity value range is [-1, 1]. The closer the value is to 1, the more consistent the two appearance features are in the embedding space and the higher the visual similarity. Based on this, as shown in Formula 1 below, the cosine similarity value of the current frame can be calculated. These are directly identified as characteristic values ​​of appearance changes. Among them, This represents the appearance feature vector of the package to be associated. Represents the appearance feature vector of candidate associated packages (Formula 1) Alternatively, to more sensitively capture mutations, as shown in Formula 2 below, the current similarity (i.e. Similarity to historical moving average (i.e.) The absolute value of the difference between the two values ​​is taken as the characteristic value of the appearance change. This indicates that cosine similarity is used for calculation.

[0030] (Formula 2) Understandably, the appearance change characteristic value can be considered a quantitative indicator used to measure the degree of difference between the current observed appearance and the historical appearance. The lower the appearance change characteristic value, the more drastic the appearance change, and the more likely the observed surface has changed due to tumbling. Taking the method shown in Formula 1 as an example, the current similarity... When the value drops sharply below a certain low threshold, it indicates that the observation surface of the package has undergone a fundamental shift.

[0031] In step S202 of some embodiments, the detection of morphological changes is described. Morphological changes can be quantified from two levels: point cloud spatial morphology and two-dimensional projection morphology. For point cloud morphological changes, the degree of change in the geometric structure of the three-dimensional spatial point set can be referred to, which can be quantified by ICP registration residuals. Specifically, firstly, the three-dimensional point cloud data corresponding to the wrapper to be associated in the current frame image to be detected can be obtained. This data is synchronously acquired by the 3D camera and provides a spatial point set in the world coordinate system. Then, the point cloud of the current frame can be ICP (Iterative Closest Point) registered with the point cloud of the candidate wrapper in the historical frame (such as the previous frame or the template point cloud at the time of registration). As shown in Equation 3 below, this registration process can iteratively optimize a rigid transformation (rotation matrix R and translation vector t) so that the two point clouds achieve the best alignment in space, and output the average Euclidean distance between all corresponding point pairs after registration as the registration score (ICP_Score).

[0032] (Formula 3) in, This represents the i-th point in the current frame. This represents the corresponding point in the historical frame, and N represents the number of matched point pairs. A higher registration score indicates a greater difference in the shape of the point cloud, meaning that the physical pose of the wrapped object has changed significantly.

[0033] Regarding changes in the shape of the detection box, these changes refer to the degree of alteration in the projected shape of the two-dimensional image, quantified by the difference in the aspect ratio of the detection box. Specifically, the detection box output by the target detection model in step S101 is first read. This detection box is defined by the coordinates of its upper left and lower right corners. As shown in Formula 4 below, the system calculates the aspect ratio of the detection box in the current frame (…). ) and the aspect ratio of the candidate associated wrapper in the previous frame ( The absolute value of the difference between the two is taken as the change in the aspect ratio of the detection box (Δaspect).

[0034] (Formula 4) in, The aspect ratio of the detection box is used. Changes in the aspect ratio reflect the degree of deformation of the package's projected outline. For example, when a package rolls around a horizontal axis, its projected height changes significantly, causing drastic fluctuations in the aspect ratio. It is understandable that point cloud registration scores or aspect ratio changes alone can be used, or both can be calculated, with either exceeding a threshold considered a morphological change condition met.

[0035] It is understandable that tumbling around different axes will result in different aspect ratio changes: around the x-axis (tumbling left and right): the height and width remain unchanged, but the shape may change due to perspective; around the y-axis (tumbling forward and backward): the height changes significantly; around the z-axis (rotation): the aspect ratio remains basically unchanged, but the point cloud distribution changes.

[0036] In step S203 of some embodiments, appearance change feature values ​​(such as...) are obtained respectively. ), positional deviation (which has been calculated in step S103, i.e., the Euclidean or Mahalanobis distance between the actual detection position and the Kalman predicted position), and point cloud morphology changes in step S202 (i.e. After detecting changes in the bounding box shape (Δaspect), a comprehensive judgment can be made based on these indicators to generate the final roll detection result. Specifically, the indicators of the above three dimensions can be compared with their respective preset thresholds: First, determine the positional deviation (denoted as Δaspect). Is it less than the position continuity threshold (denoted as)? (e.g., set to 0.3 meters) to ensure that the current target has not changed position, thus eliminating the interference scenario of "old target disappearing and new target appearing". Secondly, determine the appearance change characteristic value (e.g. Is it less than the appearance mutation threshold (denoted as)? (e.g., set to, for example, 0.3), to confirm that a fundamental shift has indeed occurred in the observation surface. Finally, determine the changes in point cloud morphology (i.e., Is it greater than the morphological change threshold (denoted as...)? (e.g., set to 0.05), or whether the change in the bounding box shape (i.e., Δaspect) is greater than the bounding box change threshold (denoted as Δaspect). (e.g., set to 0.2) to confirm that the physical posture or projected shape of the target has changed significantly. As shown in Formula 5 below, when all the above conditions (position continuity condition, appearance abrupt change condition, and significant shape change condition) are met at the same time, a rollover detection result (IsFlip = True) indicating that the package to be associated has rolled over is generated.

[0037] (Formula 5) Conversely, if any condition is not met, it is determined that no roll has occurred.

[0038] Understandably, positional deviation serves as a key constraint for excluding "new targets" here, because newly entering packages typically appear from the image boundary, with their positions deviating significantly from the predicted positions. In contrast, the positions of tumbling packages are continuous, and this constraint effectively distinguishes tumbling packages from new targets. Specifically, Table 1 below provides an example of feature differentiation between tumbling packages and new target packages.

[0039] Table 1. Examples of features distinguishing tumbled packages from new target packages.

[0040] The embodiments of this application, through the above-mentioned multi-condition joint determination, can identify tumble events with high confidence, while effectively avoiding false triggers caused by changes in lighting, partial occlusion, or detection noise, thus providing accurate status input for subsequent tumble processing.

[0041] In step S105 of some embodiments, after determining that the package to be associated has rolled, the movement speed v of the package to be associated can be compared with a corresponding preset threshold (such as a re-identification speed threshold, denoted as v). (Compare with) the re-identification speed threshold. The speed threshold that distinguishes between rapid and normal motion is different. The re-identification speed threshold is usually set to 0.5 times the speed threshold (i.e., ...). Re-identification speed coefficient This can be set to 0.5, aiming to provide sufficiently clear imaging conditions for appearance re-identification (because re-identification requires clear appearance features and thus higher imaging quality). If the speed of the package to be associated is less than the re-identification speed threshold (i.e., v < 0.5), the re-identification speed threshold is determined. This indicates that the package to be associated is moving slowly or is approaching stillness, with low imaging blur and reliable appearance features. At this point, the system triggers a refined appearance matching process. In this process, the system performs a comprehensive feature comparison between the observed appearance features of the package to be associated in the current frame's image (including cropped and aligned RGB image features and local point cloud geometric features) and a pre-built and stored feature library for candidate packages. The feature library is a database structure with the package TrackID as the primary key, storing a set of features extracted after multi-angle virtual rendering of the package's 3D image during offline or preprocessing stages. The preset viewpoint is a combination of three-dimensional spatial angles defined by discretizing and sampling the possible orientation poses of the package. That is, the feature library does not only store the appearance from a single historical viewpoint, but also includes appearance feature vectors of the candidate package under multiple preset viewpoints pre-rendered during the main imaging stage, such as ReID features and point cloud features from 20 basic viewpoints. During the comparison process, the system performs parallel database lookup operations, simultaneously calculating the similarity between the appearance features of the package to be associated and the features of all viewpoints in the feature library, and taking the highest score or weighted fusion score as the final appearance feature similarity output.

[0042] In one specific embodiment, Table 2 below shows an example table of the impact of speed on image quality.

[0043] Table 2. Examples of the impact of speed on image quality

[0044] As shown in Table 2, when the package is in a low-speed movement state (i.e., the movement speed v is lower than the re-identification speed threshold), When the package is moving at a moderate speed, the RGB image captured by the camera is clear and the point cloud data is complete. At this time, the appearance features have high reliability, making it the optimal time to perform fine-grained appearance matching. When the package is moving at a moderate speed, the image quality decreases slightly but remains basically usable, and the reliability of the appearance features is moderate. The system can selectively supplement appearance information based on the actual situation. When the package is moving at a high speed (i.e., the speed v is greater than the re-identification speed threshold), the image quality deteriorates. When the image quality degrades significantly, RGB images exhibit motion blur and trailing, and point cloud data becomes sparse and distorted, leading to a low level of reliability for appearance features. Therefore, it is not advisable to rely on appearance features for matching. Based on the velocity-quality mapping relationship shown in Table 2, this embodiment, after determining that the package to be associated has rolled over, first determines the current velocity of the package. Matching is only performed when the current velocity drops below the re-identification velocity threshold. Multi-view appearance re-identification based on a pre-rendered feature library is only triggered in the low-speed range, while in the high-speed range (i.e., v> It prioritizes motion features to maintain tracking, thereby ensuring that appearance matching is always performed when imaging conditions are optimal, fundamentally avoiding mismatches caused by image degradation.

[0045] In step S106 of some embodiments, after obtaining motion feature similarity and appearance feature similarity respectively, these two similarities can be weighted and fused to generate a comprehensive evaluation index, namely the first association score. The first association score is a comprehensive value used to finally determine whether two packages (i.e., the package to be associated and the candidate package to be associated) are the same target. Its value range is normalized to the interval [0,1], and the higher the score, the stronger the association confidence. It can be understood that the fusion weights (i.e., the motion feature weights corresponding to the motion feature similarity and the appearance feature weights corresponding to the appearance feature similarity) can be dynamically adjusted according to the current tracking state of the package to be associated. For example, as shown in Table 3 below.

[0046] Table 3 Example of Weight Adjustment

[0047] In Table 3, when the package to be associated is in a normal tracking state (e.g., when tumbling detection indicates no tumbling has occurred), the weights can be adaptively determined using a multi-feature adaptive fusion matching method. For example, the weights of motion features and appearance features can be adaptively determined based on two dimensions: the movement speed of the package to be associated and the proportion of occlusion by adjacent packages. In this case, appearance features are given a higher weight because the imaging is clear and reliable, while motion features serve as an auxiliary reference. When the package tumbles and its current movement speed is high, and it is in a "to be re-identified" state (this state will be described in detail later), the appearance features are unreliable due to blurred imaging. The system completely skips appearance feature matching (i.e., the weight is set to zero) and uses motion features (i.e., motion similarity based on Kalman filtering) as the dominant or even sole basis for matching scores, relying solely on motion prediction to maintain tracking. When the package speed decreases below the re-identification speed threshold and enters the "re-identification triggered" state (i.e., the state described in steps S104 and S105), the system resumes normal feature fusion. At this time, both appearance features (including multi-view feature library matching results) and motion features participate in joint matching, but motion features are used as an auxiliary feature, and appearance features regain their normal weight to ensure high-confidence identity verification is completed at the optimal imaging time. It is understandable that the weight configuration in the above three states is not a mutually exclusive fixed level, but can be dynamically switched according to the real-time judgment of tumbling detection and motion speed in steps S104 and S105. This ensures that the fusion matching strategy is always synchronized with the physical motion state of the package, achieving a smooth transition from "high-speed motion prediction-dependent" to "medium-low speed appearance matching" and then to "normal fusion tracking," thereby maximizing matching accuracy while ensuring tracking continuity.

[0048] In step S107 of some embodiments, the first association score is compared with a preset association threshold (the value is not limited). If the first association score is determined to be not less than the association threshold, it indicates that the current package to be associated and the candidate package to be associated have a high degree of consistency in motion trajectory, appearance and posture matching, that is, it is confirmed that the two are the same physical package, and the identity of the package to be associated is inherited as the TrackID of the candidate package to be associated, thereby completing the association operation.

[0049] Reference Figure 3 In some embodiments, the method for constructing the feature library may include, but is not limited to, steps S301 to S304.

[0050] Step S301: Obtain the 3D image of the candidate associated package acquired during the main line acquisition phase. The 3D image includes point cloud data and texture mapping data of the five faces of the candidate associated package excluding the bottom face. Step S302: Render the point cloud data and texture mapping data of each face in the 3D image at multiple preset pose angles to obtain the rendered image and rendered point cloud of the candidate association wrapped in multiple viewpoints. Step S303: Extract features from each rendered image to obtain texture features from the corresponding viewpoint, and extract features from each rendered point cloud to obtain geometric features from the corresponding viewpoint. Step S304: Associate and store each viewpoint, as well as the texture features and geometric features corresponding to each viewpoint, to construct a feature library of candidate associated wrappers.

[0051] In step S301 of some embodiments, a feature library is constructed based on the high-quality 3D image data collected by the candidate associated package during the main line sorting stage. The main line acquisition stage can refer to a specific station where data acquisition is completed when the package passes through the main line six-sided scanner and 3D imaging equipment before entering the sorting chute. Specifically, when the candidate associated package passes through the six-sided scanner and 3D imaging system on the sorting main line, the system acquires complete 3D information of the package's five sides (front, back, left, right, and top) excluding the bottom surface. This information collectively constitutes a 3D image. The 3D image contains two types of information: the first type is point cloud data, which is a set of a large number of 3D spatial coordinate points (x, y, z). These points accurately describe the geometric contours and spatial structures of each surface of the candidate associated package; the second type is texture mapping data, which is RGB color information or texture maps corresponding to the point cloud coordinates, so that each 3D point is associated with a color attribute, thereby restoring the visual appearance of the package surface (such as label patterns, trademarks, printed text, etc.). It is understandable that, since the bottom surface of the candidate associated package is in contact with the conveyor belt during the main line transmission, the 3D camera cannot collect bottom surface data, so the 3D image naturally lacks bottom surface information.

[0052] In step S302 of some embodiments, after obtaining the 3D image, the multi-faceted data of the 3D image can be virtually rendered at multiple preset pose angles to generate simulated images (i.e., rendered images) and simulated point clouds (i.e., rendered point clouds) of candidate associated wrappers under various possible viewing angles. Specifically, firstly, a finite pose space set Φ can be defined, which is generated by discretizing the possible orientations of the wrapper. For example, as shown in Table 4 below, the pose space consists of two dimensions: the first dimension is "facing upwards", that is, which face of the wrapper faces the camera's field of view. This dimension contains 5 states, corresponding to front, back, left, right, and top facing upwards (since the bottom face is a missing face, it is not considered an independent facing upwards state); the second dimension is "rotation around the normal vector", that is, after determining facing upwards, the angle by which the wrapper rotates around the normal direction of that face. This dimension contains 4 states, namely 0°, 90°, 180°, and 270°.

[0053] Table 4 Example of Pose Space

[0054] By combining the above two dimensions, 5 × 4 = 20 basic viewpoints can be formed. A viewpoint refers to the relative spatial relationship between the observation direction and the orientation of the package, defined by "facing upwards" and the rotation angle around the normal vector. Furthermore, embodiments of this application can further subdivide the basic viewpoints. For example, as shown in Table 5 below, interpolated viewpoints are inserted between adjacent basic viewpoints. Subdividing each basic viewpoint into 2 parts yields 40 viewpoints (medium granularity), and subdividing it into 4 parts yields 80 viewpoints (fine granularity). To achieve a balance between matching accuracy and storage overhead, a medium-granularity scheme of 40 viewpoints can be adopted. When using a medium-granularity scheme, the viewpoint symbol can be represented as: , where N=40.

[0055] Table 5 Examples of Subdivided Perspectives

[0056] After determining multiple perspectives (e.g., 40 perspectives), for each perspective... The following rendering operations can be performed: First, the point cloud data in the 3D image is rendered according to the viewpoint. A rigid transformation (rotation and translation) is performed on the corresponding spatial pose to place the point cloud in the virtual camera coordinate system of that viewpoint. The transformed 3D point set is the rendered point cloud P_transformed = R_ P_3Dprofile + t_ Essentially, it is the original 3D image point cloud (P_3Dprofile) rotated (R_ ) and translation (t_) The point set obtained, expressed in the virtual camera coordinate system and retaining complete 3D spatial coordinate information, is used as input data for subsequent geometric feature extraction. Then, pre-calibrated virtual camera parameters (including intrinsic parameter matrix K, rotation matrix, etc.) can be used. Translation vector This process maps 3D spatial points from a rendered point cloud onto a 2D image plane using a perspective projection model, generating a rendered image from that viewpoint. The rendered image includes the rendered RGB image I_render( ) and rendering depth map D_render( The rendered RGB image records the texture color information corresponding to each 3D point in the point cloud, which is then projected to form a color image for subsequent texture feature extraction. The rendered depth map records the distance information of each 3D point in the point cloud from the optical center of the virtual camera, which is then projected to form a depth image and can be used as auxiliary information for occlusion detection or shape analysis. It can be understood that the generation of the rendered RGB image depends on the texture mapping data in the 3D image; that is, the RGB color value associated with each 3D point is assigned a corresponding 2D pixel position during projection, thus forming a color image. The rendered depth map, on the other hand, depends on the spatial depth coordinates of each 3D point. Both originate from the same rendered point cloud data projected onto different information dimensions.

[0057] In step S303 of some embodiments, after obtaining the rendered images and rendered point clouds from various viewpoints, texture features and geometric features can be extracted from these two rendering data respectively. For texture feature extraction, the rendered RGB image I_render( generated in step S302) is used... The input is fed into a pre-trained ReID network. The ReID network outputs a high-dimensional RGB feature vector: The RGB feature vector It encodes the appearance information of the wrapped surface in the image, such as color distribution, pattern texture, and text markings. After extracting the RGB feature vector... Then, as shown in Formula 5 below, L2 norm normalization can be performed to eliminate the influence of modulus differences on similarity calculation. The normalized feature vector is the viewpoint. Texture features below Texture features are deep embedding vectors that describe the visual appearance of a package surface, reflecting appearance attributes such as color, texture, and pattern, and are highly distinguishable in identifying packages.

[0058] (Formula 5) For geometric feature extraction, the rendered point cloud P_transformed obtained after pose transformation in step S302 can be input into the point cloud feature extraction network. This network can be PointNet or other deep learning architectures specifically designed for processing unordered point clouds. The point cloud feature extraction network outputs a fixed-dimensional geometric feature vector: The geometric feature vector It encodes geometric morphological information such as the three-dimensional contour, size ratio, and surface curvature wrapped in this viewpoint. Similarly, as shown in Formula 6 below, this geometric feature vector... L2 norm normalization is performed to obtain geometric features. Geometric features are deep embedding vectors that describe the three-dimensional morphological structure, reflecting spatial attributes such as size, shape, and curvature, and can still provide discriminative information when textures degrade.

[0059] (Formula 6) In step S304 of some embodiments, after extracting texture and geometric features from all viewpoints (e.g., 40 viewpoints), these data can be stored in a structured manner to construct a complete feature library of candidate associated wrappers. Specifically, a database entry or memory-mapped table with the wrapper TrackID as the primary key can be created, and viewpoint indexes can be used under this entry. For a second-order bond, the first A logo from a different perspective The corresponding texture feature vector f_RGB( ), and the corresponding geometric eigenvector g( These three elements are stored as a single record in the table. Furthermore, to support efficient parallel database lookup operations, the feature library can employ a key-value structure or a columnar storage format. This allows for the loading of all viewpoint features for the same package at once during the matching phase, and the batch calculation of the similarity between the currently observed feature and all viewpoint features using vectorized computing units (such as GPUs or SIMD instruction sets). After construction, the feature library is persistently stored in system memory or a cache, bound to the package's TrackID, and released when the package leaves the tracking area or is determined to be permanently lost (Deleted).

[0060] Reference Figure 4 In some embodiments, appearance features include texture features and geometric features. In step S105, “comparing the appearance features to be associated and wrapped in the image to be detected with the feature library of candidate associated wrappers to determine the appearance feature similarity” may include, but is not limited to, steps S401 to S403.

[0061] Step S401: Obtain the historical pose set of candidate associated packages. The historical pose set includes viewpoints selected from the feature library corresponding to the candidate associated packages based on their appearance features in the previous frame or multiple previous frames. Step S402: If the distribution of each viewpoint in the historical pose set is the target distribution, the texture features and geometric features to be associated and wrapped are compared with the texture features and geometric features corresponding to each viewpoint in the historical pose set to obtain the first appearance matching score of each viewpoint. Step S403: Remove the viewpoints in the historical pose set whose first appearance matching score is less than a preset threshold. If the historical pose set is not empty after removal, determine the maximum score in the historical pose set after removal as the appearance feature similarity.

[0062] In step S401 of some embodiments, a historical pose set of the candidate associated package is obtained. This historical pose set is not a three-dimensional position coordinate or attitude angle of the package in physical space, but rather one or more viewpoint identifiers selected from the feature library of the candidate associated package during the matching processing cycle of the previous frame or multiple previous frames. The set, where each viewpoint identifier corresponds to a pre-stored appearance feature (including texture feature vector f_RGB) when viewed from a specific orientation. ) and geometric eigenvectors g( Specifically, the historical pose set is generated in two ways: First, through active parallel matching: When processing the image of the previous frame (i.e., time t-1), the system extracts the observed texture features and observed geometric features of the candidate associated package in the current frame (frame t-1), compares them one by one with the pre-stored appearance features of all preset viewpoints (e.g., 40) in the feature library, and filters out the viewpoint set that is higher than the preset threshold based on the matching score. This set is cached as the matching result at time t-1. Second, through inheriting the pose set of the previous frame: In the processing at time t-1, the system does not re-traverse and match all preset viewpoints, but directly uses the pose set retained after matching verification at an earlier time (e.g., time t-2 or several previous frames), and updates it by removing and selecting the best ones. The updated set is cached as the historical pose set at time t-1. In other words, the historical pose set is essentially a recursive quantity that is continuously passed along the timeline: the historical pose set may be recalculated in each frame (generating a new set after parallel matching of the full feature library), or it may be obtained by inheriting the set of the previous frame and modifying it (a fast inheritance strategy in high-confidence scenarios).

[0063] In step S402 of some embodiments, after obtaining the historical pose set, it can be determined whether the distribution of each viewpoint in the historical pose set meets the target distribution condition. If it does, a refined feature comparison is performed directly within the set. Specifically, as shown in Table 6 below, the target distribution refers to a high-confidence unimodal distribution state, that is, the matching score of a certain viewpoint in the historical pose set is significantly higher than all other viewpoints (e.g., the difference between the highest score and the second highest score exceeds a preset threshold). In this case, the system determines that the pose of the package is highly consistent with the viewpoint, and determines that the pose of the package is the orientation corresponding to the viewpoint. At the same time, it also indicates that the matching result of the previous frame (t-1) is highly deterministic, and the pose of the package will not change abruptly between frames. Therefore, the pose of the current frame (t) should be similar to that of the previous frame, and there is no need to retraceive all the viewpoints in the feature library.

[0064] Table 6 Example of score distribution

[0065] As shown in Table 6, when the score distribution exhibits a multi-peak pattern, meaning that multiple adjacent viewpoints (e.g., facing upwards with the same angle and adjacent rotation angles, or facing upwards with the same angle and adjacent rotation angles) have relatively close scores and are all at a high level, the system determines that the package posture is in a transitional state between these viewpoints. When the score distribution exhibits a uniform pattern, meaning that the scores of all preset viewpoints are at a low and similar level with no significant peaks, the system determines that the current posture cannot be effectively matched with any preset viewpoint in the feature library. This situation usually corresponds to the package's bottom surface facing outwards (because the feature library does not contain a bottom surface viewpoint) or the posture being extremely uncertain.

[0066] After determining that the distribution of each viewpoint in the historical pose set satisfies the target distribution, the observed texture features (i.e., RGB appearance embedding vectors extracted and normalized from the cropped image of the image to be detected through the ReID network) and observed geometric features (i.e., 3D shape embedding vectors extracted and normalized from the segmented point cloud of the image to be detected through the point cloud network) of the current frame (t) to be associated can be extracted. Simultaneously, the texture and geometric features corresponding to each viewpoint in the historical pose set are retrieved from the feature library of the candidate associated wrapping. Then, the cosine similarity of the observed texture features of the current frame with the pre-stored texture features of each viewpoint in the historical pose set is calculated to obtain the texture similarity component of each viewpoint. Simultaneously, the feature distance of the observed geometric features of the current frame with the pre-stored geometric features of each viewpoint in the historical pose set is calculated and mapped to similarity to obtain the geometric similarity component of each viewpoint. Further, the above two components can be fused to obtain the first appearance matching score for each viewpoint. The higher the first appearance matching score, the better the matching degree between the current observation and that viewpoint.

[0067] In step S403 of some embodiments, after calculating the first appearance matching score for each viewpoint in the historical pose set, the historical pose set can be refined and filtered, and the appearance feature similarity of the current frame can be determined based on the highest score after filtering. Specifically, a preset score threshold (denoted as...) is set... (The value is not limited) is used as a retention condition, and the first appearance matching score of each viewpoint in the historical pose set is judged one by one to see if it is lower than the threshold. If the first appearance matching score of a certain viewpoint is lower than the score threshold... This indicates that the observed appearance of the current frame no longer has sufficient consistency with the pre-stored appearance features of that viewpoint, and the pose assumption represented by that viewpoint is no longer valid. Therefore, the viewpoint is removed from the historical pose set. Further, it is checked whether the historical pose set after the above removal operation is empty. If the set is not empty after removal, it indicates that at least one viewpoint can still obtain a sufficiently high matching score in the current frame. Thus, the viewpoint with the highest first appearance matching score is selected from the retained viewpoints, and its score is directly determined as the appearance feature similarity between the package to be associated and the candidate associated package. It is understandable that this removed historical pose set can be cached as the historical pose set of the current frame for inheritance during matching in the next frame (t+1).

[0068] As described above, in scenarios where the historical pose set is the target distribution, the method for determining the similarity of appearance features includes the following steps: Step 1: Inherit the historical pose set Φ{t-1} from the previous frame; Step 2: For each pose i (i.e., viewpoint) in the historical pose set Φ{t-1} Perform the following operations: 2.1 Calculate a new matching score Si,t (i.e., the first appearance matching score) with the appearance features of the package to be associated observed in the current frame. 2.2 If the new matching score Si,t < score threshold Remove pose i.

[0069] Step 3: If the historical pose set Φ{t-1} is not empty after the removal operation, determine the appearance feature similarity based on the maximum matching score.

[0070] Understandably, in conventional tracking scenarios (without tumbling), the orientation of the package remains largely unchanged between adjacent frames. Therefore, conventional methods that directly calculate the cosine similarity between the observed appearance features of the current frame and the appearance features of the candidate associated package in historical frames (such as the previous frame) to determine the similarity of appearance features are reasonable and efficient. However, in tumbling scenarios, the orientation of the package changes fundamentally. For example, the current frame might observe the side or top of the package, while the previous frame might have observed the front. These are different surfaces of the package, and even if they belong to the same physical package, their appearance features will inevitably differ significantly in the embedding space. In this case, directly comparing them using conventional methods will lead to a sharp drop in similarity, making correct association impossible. To address this, this application introduces a pre-rendered multi-view feature library, transforming the comparison object from "the observed appearance of historical frames" to "the pre-stored appearance features of the package's 3D image under multiple preset views." This ensures that regardless of which side the package presents after tumbling, the system can find the view that best matches the observation from the feature library for comparison. Building upon this, steps S401 to S403 further utilize the prior knowledge of inter-frame pose continuity, namely, that pose changes encapsulated between adjacent frames are small and gradual. Therefore, when the historical pose set exhibits a high-confidence distribution, the optimal viewpoint of the current frame must fall near this set, eliminating the need to re-traverse all views in the feature library. In summary, the method provided by steps S401 to S403 not only fundamentally solves the viewpoint mismatch problem caused by rolling but also reduces the appearance matching search space of the current frame from all views in the feature library to a small number of views in the historical pose set, significantly reducing computational overhead.

[0071] In some embodiments, step S105, "comparing the appearance features of the package to be associated in the image to be detected with the feature library of candidate associated packages to determine the appearance feature similarity", may further include the following steps: If the set of historical poses is empty after removal, the geometric features to be associated wrapped in the image to be detected are compared with the geometric features of the candidate to be associated wrapped in the previous frame image to obtain the appearance feature similarity.

[0072] In this embodiment, after step S403 removes the historical pose set, if the historical pose set is empty after removal, it indicates that the high-confidence pose assumption of the previous frame is completely invalid in the current frame, meaning that the pose of the package has changed significantly and exceeds the range covered by the historical viewpoint. At this time, it is considered that the package to be associated may be in a bottom-facing state, that is, the package surface currently observed by the camera is the bottom surface that was not collected during the main line imaging. This surface has no corresponding pre-stored viewpoint features in the feature library. At this time, the system cannot rely on appearance features for reliable matching and needs to switch to a motion feature-driven tracking mode. To this end, this embodiment provides two optional implementation methods. Both methods are based on motion features and differ only in their specific implementation methods. The first method, as shown in Table 7 below, uses a weight redistribution method for fusion matching: the weight of the RGB texture feature is set to zero, that is, the RGB appearance matching of the current frame is completely skipped to avoid introducing noise due to the lack of texture information on the bottom surface. Geometric features (i.e., point cloud shape features) are given a low weight (e.g., 0.2) because although point clouds can be collected when the bottom surface is facing outwards, there is a lack of corresponding historical bottom surface comparisons, resulting in limited discriminative power. Motion features (i.e., motion feature similarity) are given a dominant weight (e.g., 0.8), meaning that the motion similarity predicted by the Kalman filter is used as the absolute subject of the comprehensive matching score, thereby maintaining the continuity of the tracking ID.

[0073] Table 7 Example of Weight Allocation

[0074] In other words, appearance feature similarity is determined solely based on geometric feature similarity. Specifically, firstly, the observed geometric features of the package to be associated in the image to be detected are obtained, i.e., the 3D shape embedding vectors extracted from the package point cloud segmented from the current frame by a point cloud feature extraction network. Simultaneously, the historical geometric features corresponding to the candidate associated package in the previous frame are obtained. Then, the feature distance (e.g., Euclidean distance) between the observed geometric features of the current frame and the historical geometric features of the previous frame is calculated, and this distance is converted into a similarity score through exponential mapping or normalization, serving as the appearance feature similarity output for the current frame.

[0075] The second approach is more direct. The system completely abandons any form of appearance matching attempt in the current frame, relying solely on the kinematic predictions of the Kalman filter to maintain package tracking. Specifically, it multiplies the state transition matrix F with the state estimate from the previous frame to obtain the prior state prediction for the current frame. This prediction includes the target's expected position (i.e., predicted position) and velocity information in the current frame. In this mode, the package's TrackID remains unchanged; the system only outputs the predicted position as the tracking result for the current frame, without binding it to any detection results for confirmation.

[0076] The first and second methods described above are parallel preferred embodiments for scenarios where the bottom surface faces outwards. The choice between the two can be flexible depending on the actual application scenario: the first method retains low-weight geometric feature constraints, suitable for scenarios where the bottom surface point cloud quality is acceptable, and can provide a certain degree of shape verification; the second method has lower computational requirements, suitable for extreme scenarios where no reliable appearance information can be obtained at all. Regardless of the strategy adopted, the core idea remains the same: in the blind spot of the missing bottom surface, motion features are used as the primary means to ensure that the package identity is not lost, avoiding erroneous deletion or re-registration of the target due to appearance matching failure.

[0077] Understandably, in another embodiment, after removing the historical pose set in step S403, if the set is empty after removal, the system does not immediately determine that the bottom face is facing outward and switch to the motion feature-dominated mode. Instead, it first executes an expanded search strategy in the view space to deal with situations where the wrapping pose has undergone a moderate change but has not yet exceeded the overall coverage of the feature library. Specifically, when the entire historical pose set becomes invalid in the current frame (i.e., in step 2.2 above, the matching score Si,t of all poses in the historical pose set is less than the score threshold), The system uses the geometric center of the historical pose set in the pose space as a reference (or the highest matching score in the historical pose set as a reference), and expands the search range to include adjacent viewpoints of the historical pose set in the view space according to a preset neighborhood expansion radius (e.g., expanding to adjacent faces in the upward dimension, and expanding ±1 offset step length in the rotation angle dimension), forming an expanded candidate viewpoint set. The matching score of each viewpoint in this candidate viewpoint set is then recalculated with the observation features of the current frame. If, after the expanded search, the matching score of any viewpoint exceeds a score threshold... If the highest score is determined as the appearance feature similarity, the pose set is updated for inheritance in the next frame. If no valid viewpoint can be matched after expanding the search, and the matching scores are still all below the score threshold, then... This indicates that the current observed posture has significantly deviated from all historical and adjacent view assumptions. Only then does the system finally confirm that the package is in a bottom-facing state and switch to a motion-feature-driven tracking strategy. Through the progressive processing logic of "historical set inheritance → removal → adjacent view expansion → bottom-facing determination" described above, the system fully utilizes the local continuity of the posture space before determining that the bottom is facing outward. This avoids prematurely abandoning appearance matching when the historical pose set becomes temporarily invalid due to slight posture deviations. Thus, while ensuring matching accuracy, it effectively reduces the false trigger rate of the motion-feature-driven mode.

[0078] Reference Figure 5 In some embodiments, the package association method provided in this application may further include steps S501 to S507.

[0079] Step S501: If the historical pose set is empty after removal, mark the package to be associated as having its bottom face facing out. Step S502: Perform state detection on the forward-tracing of the package to be associated for multiple consecutive frames, and determine whether the feature update conditions are met. The feature update conditions include: the package to be associated is marked as bottom-facing out in multiple consecutive frames, and the package to be associated is not in a tumbling transition state in the image to be detected. Step S503: If the package to be associated meets the feature update conditions, extract the bottom point cloud data of the package to be associated in the image to be detected. Step S504: Determine the bottom geometric features of the package to be associated based on the bottom point cloud data, and fuse the bottom geometric features with the 3D image of the candidate associated package to obtain the updated 3D image. Step S505: Render the updated 3D image at multiple pose angles to obtain the rendered image and rendered point cloud of the new viewpoint. Step S506: Extract features from the rendered image of the new viewpoint to obtain new texture features, and extract features from the rendered point cloud of the new viewpoint to obtain new geometric features. Step S507: The newly added viewpoint, newly added texture features, and newly added geometric features are associated and stored in the feature library of the candidate associated package.

[0080] In step S501 of some embodiments, after removing the historical pose set in step S403, if the removed historical pose set is empty, it indicates that the high-confidence viewpoint assumption of the previous frame (t-1) is completely invalid in the current frame (t), and no effective matching viewpoint can be obtained after adjacent viewpoint expansion search. At this time, the system determines that the observed surface of the package to be associated exceeds the coverage of the pre-rendered feature library. Based on the physical continuity of the package rolling and the prior knowledge that the bottom surface is the only missing surface to be acquired as the main line, the package to be associated is marked as bottom-facing outward. The bottom-facing outward state can be a Boolean state identifier, used to indicate that the bottom surface of the package to be associated is facing the camera's field of view, that is, the package surface currently observed by the system is a missing surface that was not acquired during the main line 3D imaging stage due to contact with the conveyor belt.

[0081] In step S502 of some embodiments, after marking the package to be associated as having its bottom face out, a forward-looking multi-frame state detection can be performed on the package to determine whether the preset feature update conditions are met. The forward-looking multi-frame state detection can refer to using cached historical frame tracking results to check the package's state identifier sequence frame by frame from the current frame in the historical direction. The feature update conditions are a set of admission rules that trigger 3D image and feature library update operations, used to ensure the reliability and data quality of the update operations. Specifically, the feature update conditions can be designed as a two-stage judgment logic: the first condition is a temporal persistence judgment, that is, the system backtracks the cached historical state records to confirm whether the package to be associated has been marked as having its bottom face out in N consecutive frames (N≥3, without limitation). This condition is used to eliminate occasional misjudgments of bottom face out caused by instantaneous attitude jitter or detection noise, ensuring that the bottom face is indeed stably exposed in the camera's field of view. The second condition is motion stability determination. The system determines that the package to be associated is currently in a non-tumbling transition state based on the state covariance matrix of the Kalman filter or the magnitude of positional changes between adjacent frames. A non-tumbling transition state refers to a state where the package's attitude change is relatively stable and its angular velocity is low, distinct from the violent attitude change phase during high-speed rotation and tumbling. This second condition ensures the package remains relatively stable during bottom surface information acquisition, thereby obtaining high-quality bottom surface point cloud data. Only when both conditions are met does the system determine that the package to be associated meets the feature update conditions, allowing subsequent bottom surface information extraction and feature library expansion operations to be performed.

[0082] In step S503 of some embodiments, after confirming that the package to be associated meets the feature update conditions, the point cloud data of the package's bottom surface can be extracted from the 3D point cloud data corresponding to the current frame's image to be detected. The bottom surface point cloud data can refer to the set of 3D point coordinates P_bottom = {p | p ∈ P_obs and are bottom surface points} of the bottom surface of the package to be associated. This set reflects the geometric contour and spatial structure of the bottom surface and is obtained through observation of the missing bottom surface data. Specifically, firstly, the current frame's depth image or point cloud data synchronously acquired by the 3D camera is acquired, and the target point cloud region corresponding to the package to be associated is segmented from the global point cloud based on the detection box output by the target detection model in step S101. Then, fine recognition and extraction of the bottom surface point cloud are performed within this target point cloud region, for example, based on depth gradient or geometric features.

[0083] In step S504 of some embodiments, after extracting the bottom point cloud data, the bottom geometric features of the package to be associated can be determined based on the bottom point cloud data, and the bottom geometric features are fused with the 3D image of the candidate associated package acquired on the main line to obtain an updated 3D image. Specifically, firstly, the extracted bottom point cloud data P_bottom is input into a point cloud feature extraction network (such as PointNet or PointNet++) to obtain the bottom geometric feature vector g_bottom, which encodes the overall shape, flatness, size ratio, and other geometric attributes of the bottom surface. At the same time, the estimated size of the bottom surface (including length and width) is calculated based on the spatial distribution of the bottom point cloud and retained as explicit geometric parameters. Then, a 3D image fusion operation is performed: the bottom point cloud data P_bottom observed in the current frame is transformed from the current camera coordinate system to the unified world coordinate system where the main line 3D image is located through pose estimation and coordinate transformation to obtain the aligned bottom point cloud. Furthermore, the aligned bottom point cloud is compared with the original 3D image point cloud. Perform union and fusion to form an updated 3D image. During the fusion process, point cloud registration (such as ICP) can be used to refine the alignment accuracy, and redundant points in overlapping areas can be eliminated by point cloud downsampling or voxel mesh fusion to maintain the uniformity of point cloud data.

[0084] In step S505 of some embodiments, after obtaining the updated 3D image, virtual rendering can be performed based on the updated 3D image at multiple pose angles to generate new viewpoint rendering images and rendering point clouds for expanding the feature library. Specifically, the rendering process is similar to the pre-rendering process described in step S302, that is, a set of new viewpoints Φ_new is defined, which takes the bottom facing upward as the core pose (i.e., the bottom is facing upward), and combines multiple rotation angles (such as 0°, 90°, 180°, 270°) around the bottom normal vector to form multiple new viewpoints. Then, for each new viewpoint... This allows you to transform the updated 3D image point cloud to this viewpoint. Under the corresponding spatial pose, the virtual camera parameters are used to project and generate the rendered RGB image I_render( ) and rendering depth map D_render( Simultaneously, it saves the 3D point cloud after pose transformation as the rendering point cloud P_render( ).

[0085] In step S506 of some embodiments, after obtaining the rendered image and rendered point cloud from the new viewpoint, feature extraction can be performed on the rendered image and rendered point cloud from the new viewpoint respectively to obtain new texture features and new geometric features. Specifically, for texture feature extraction, the rendered RGB image I_render( from the new viewpoint) is used to extract the new texture features. The input is fed into a pre-trained ReID network (using the same network architecture as in step S303), and after forward propagation, a high-dimensional embedding vector is output. This vector is then normalized using the L2 norm to obtain the new texture features. For geometric feature extraction, the rendered point cloud P_render( from the new viewpoint) is... The input is fed into a point cloud feature extraction network (using the same network architecture as in step S303). After forward propagation, a high-dimensional embedding vector is output and normalized using the L2 norm to obtain the new geometric features. Among them, the new texture features are deep embedding vectors describing the color and appearance information of the bottom surface under the new viewpoint, and the new geometric features are deep embedding vectors describing the three-dimensional shape structure of the bottom surface under the new viewpoint. Together, they constitute a complete appearance feature description under the new viewpoint.

[0086] In step S507 of some embodiments, a new perspective will be added. The corresponding feature vectors are associated and stored in the feature library of the candidate associated package, thus completing the dynamic expansion of the feature library. Specifically, using the TrackID of the candidate associated package as the primary key, a new entry is added to the data structure of its feature library. This entry contains the identifier of the new viewpoint. (For example, "bottom surface_0°", "bottom surface_90°", etc.), and corresponding newly added texture features and geometric features. Simultaneously, the system marks this new entry as "updated" to distinguish it from the original viewpoint features built during the main line pre-rendering stage. At this point, the feature library for this candidate associated package has expanded from the original 40 preset viewpoints to 48 (or more) viewpoints, achieving complete coverage of bottom surface information. Subsequently, when the package rolls over again and exposes its bottom surface, the pre-stored features corresponding to the bottom surface viewpoint can be directly retrieved from the feature library, significantly improving the efficiency and success rate of re-identification.

[0087] Reference Figure 6 In some embodiments, the step S105, which involves “comparing the appearance features of the package to be associated in the image to be detected with the feature library of candidate associated packages to determine the appearance feature similarity”, may also include, but is not limited to, steps S601 to S602.

[0088] Step S601: If the distribution of each viewpoint in the historical pose set is a non-target distribution, the appearance features of the package to be associated are compared with the texture features and geometric features corresponding to each viewpoint in the feature library corresponding to the candidate associated package to obtain the second appearance matching score of each viewpoint. Step S602: Construct the current pose set for each viewpoint whose second appearance matching score is greater than a preset threshold, and determine the maximum score in the current pose set as the appearance feature similarity.

[0089] In step S601 of some embodiments, when it is determined that the distribution of each viewpoint in the historical pose set of the candidate associated package is a non-target distribution (e.g., the matching score distribution of the previous frame as shown in Table 6 shows a multi-peak shape, and the scores of each viewpoint are relatively uniform), it indicates that the pose estimation result of the previous frame lacks sufficient reference value in the current frame and cannot be quickly matched using the historical pose inheritance strategy. At this time, the full-database parallel matching process is triggered. Specifically, firstly, the observed appearance features of the package to be associated in the current frame are compared with the pre-stored appearance features of all viewpoints in the feature library corresponding to the candidate associated package, and then compared viewpoint by viewpoint. Among them, the observed appearance features include observed texture features. (i.e., the RGB appearance embedding vector extracted and normalized from the cropped image of the image to be detected by the ReID network) and observed geometric features. (That is, extracting and normalizing a 3D shape embedding vector from the point cloud segmentation of the image to be detected through a point cloud network). For the feature library index is Each perspective First, retrieve the pre-stored texture features corresponding to that viewpoint from the feature library. and pre-stored geometric features Then, the similarity components of the two dimensions are calculated: RGB similarity. Cosine similarity can be used as a metric, i.e. , A value closer to 1 indicates a more consistent texture appearance. Geometric similarity An exponential mapping based on feature distance can be used, i.e. ,in, The scale parameter indicates that the smaller the distance, the higher the similarity. The closer it gets to 1.

[0090] After obtaining the two similarity components, a weighted sum can be performed based on the preset fusion weights to obtain the perspective. The overall matching score (i.e., the second appearance matching score). .in and These are the texture feature weights and geometric feature weights, respectively, and their sum is normalized to 1, for example... =0.6, =0.4, no limit is imposed on this.

[0091] The above calculation is performed on all preset viewpoints (e.g., 40 viewpoints) in the feature library one by one, and finally the second appearance matching score corresponding to each viewpoint is obtained.

[0092] In step S602 of some embodiments, a preset pose screening threshold (denoted as...) is used. The algorithm filters all viewpoints, constructs a candidate pose set for the current frame from viewpoints that meet the score criteria, and selects the highest score from this set as the appearance feature similarity output. Specifically, first, the score vectors [S_1, S_2, ..., S_N] of all viewpoints are sorted from highest to lowest score, forming a descending score sequence. Then, the second appearance matching score of each viewpoint is compared with the pose determination threshold. Comparison, pose determination threshold It is a value between 0 and 1 (e.g., it can be set to 0.5, there is no limitation on this). For scores greater than the pose determination threshold... The system identifies the viewpoints with scores less than or equal to θ_pose as valid candidate viewpoints and adds them to the candidate pose set Φ_candidate for the current frame. Viewpoints with scores less than or equal to θ_pose are removed as low-confidence hypotheses that do not hold true in the current frame. After the above filtering, the system checks the state of the candidate pose set: if the set is not empty, it indicates that at least one viewpoint has sufficient consistency with the current observation. Thus, the viewpoint with the highest second appearance matching score can be selected from the set, and its score is determined as the appearance feature similarity of the current frame, which is used for adaptive weighted fusion with motion feature similarity in step S106. At the same time, the candidate pose set Φ_candidate = {i | S_i>θ_pose} will be cached and bound to the TrackID of the package to be associated, serving as the historical pose set that can be inherited when matching in the next frame (t+1). If the set is empty, it indicates that the current observation has not obtained sufficient matching scores under all preset viewpoints, and the system will trigger the aforementioned adjacent viewpoint expansion search or bottom-facing-out determination process.

[0093] The embodiments of this application, through the complete process of full database traversal, score sorting, threshold filtering, and selection of the highest score, can comprehensively cover all pose assumptions in non-target distribution scenarios, determine appearance feature similarity in a globally optimal manner, and ensure that reliable matching basis can be obtained under any pose condition.

[0094] In some embodiments, the package association method provided in this application may further include the following steps: If the movement speed of the package to be associated is not less than a preset threshold, the second association score between the package to be associated and the candidate associated packages is determined based on the similarity of movement features. If the second association score is not less than the preset threshold, the package to be associated will be associated with the candidate associated package.

[0095] In this embodiment of the application, when it is determined according to step S105 that the movement speed of the package to be associated is not less than a preset threshold (i.e., satisfying v≥...), When the exposure time is insufficient relative to the speed of movement, it indicates that the package to be associated is currently in high-speed motion. In this state, as shown in Table 2, the RGB images captured by the camera exhibit motion blur and trailing due to insufficient exposure time relative to the speed of movement. Point cloud data becomes sparse and distorted due to scanning frequency limitations, significantly reducing the reliability of appearance features. Therefore, the system should not rely on any form of appearance matching. Based on this, the system enters a motion feature-dominated tracking mode, completely skipping the comparison of RGB texture features and geometric features, and directly determining the second association score between the package to be associated and the candidate associated packages based on motion feature similarity. Specifically, the system directly uses this motion feature similarity (i.e., the similarity value obtained by mapping the predicted position based on Kalman filtering to the actual detection position using Mahalanobis or Euclidean distance in step S103) as the second association score. At this time, the appearance feature weight in the fusion weight is set to zero, and the motion feature weight is set to 1.

[0096] After determining the second association score, the score can be compared with the preset association threshold: if the second association score is not less than the association threshold, it indicates that even when the appearance information is completely unavailable, the package to be associated and the candidate associated package still have a high degree of consistency in the motion trajectory (e.g., continuous position, velocity vector matching). At this time, the system will associate the package to be associated with the candidate associated package while keeping the TrackID unchanged.

[0097] Understandably, in this high-speed mode, the Kalman filter state prediction used by the system is based on the extended state vector. This state vector includes not only the target's position and velocity components in three-dimensional space, but also attitude angle components (i.e., the rotation angle of the wrap around the three coordinate axes) to accommodate the initial estimate of the wrap's orientation change during tumbling. Its state prediction equation is: ,in, Let F be the prior state estimate at time t (i.e., the state at the current time is predicted based on the posterior estimate at time t-1), and let F be the state transition matrix. B is the posterior estimate at time t-1, and B is the control input matrix (describing how control quantities such as appearance acceleration or angular velocity affect state changes). Let be the control vector (representing the external input acting on the package). Through the above-described pure kinematic prediction and matching mechanism, the system can maintain tracking continuity even under high-speed imaging degradation conditions, until the package speed decreases to below 100 km / h. Then, the system switches back to the normal appearance + motion matching mode for identity verification and confirmation.

[0098] The package association method provided in this application achieves significant overall benefits by constructing a complete technical closed loop of "pre-rendered feature library - roll detection - speed perception - multi-strategy matching - bottom self-evolution". First, by rendering the 3D image of the package collected in the main stage from multiple perspectives and extracting texture and geometric features to construct a feature library, the re-identification operation after roll is reduced from real-time 3D rendering calculation to efficient vector database comparison, which greatly reduces online processing latency and meets the real-time requirements of industrial sorting scenarios. At the same time, the comprehensive coverage of multiple perspectives in the feature library ensures that the package can find a corresponding matching template in any posture, fundamentally solving the problem of matching failure when the posture changes in traditional single-view methods. Second, roll detection is achieved by integrating the 3D joint judgment logic of appearance mutation, position continuity and morphological change. Combined with speed perception re-identification timing control (pure motion prediction to maintain tracking at high speed, and triggering appearance re-identification at medium and low speed), unreliable matching under imaging degradation conditions is effectively avoided, ensuring both tracking continuity and re-identification accuracy. Furthermore, by employing an adaptive strategy that combines historical pose set inheritance matching under high confidence with full-database parallel matching under low confidence, computational efficiency is significantly optimized while ensuring matching accuracy. A smooth transition mechanism is provided when historical poses become completely invalid through adjacent viewpoint expansion search and degraded geometric matching, enabling the system to operate stably across the entire confidence range. In addition, after continuous multi-frame stability assessment, the system utilizes observation data of the bottom surface accidentally exposed by the package to complete the 3D image and dynamically expand the feature library, giving the system online self-evolution capabilities: as the running time increases, the feature library covers increasingly complete viewpoints, and the success rate of subsequent re-identification continuously improves, forming a virtuous cycle of positive feedback. In summary, the method provided in this application systematically solves the problem of identity fragmentation caused by fundamental changes in the observation surface in package tumbling scenarios, significantly improving the robustness, real-time performance, and adaptability of the logistics sorting and tracking system in complex physical environments.

[0099] Reference Figure 7 This application also provides a package association device for package tracking within a single camera. The device includes: The package detection unit 701 is used to acquire the image to be detected captured by the camera, and to perform package detection on the image to be detected to determine at least one package to be associated in the image to be detected. The candidate association determination unit 702 is used to determine, for each package to be associated, a candidate associated package corresponding to the package to be associated from the tracked packages contained in the previous frame image corresponding to the image to be detected, captured by the camera. The position deviation determination unit 703 is used to predict the position of the candidate associated package in the image to be detected to obtain the predicted position, and to determine the motion feature similarity based on the position deviation between the actual detected position and the predicted position of the package in the image to be detected. The tumbling detection unit 704 is used to perform tumbling detection on the package to be associated. If the tumbling detection result indicates that the package to be associated has tumbled, the unit obtains the motion speed of the package to be associated in the image to be detected. The appearance comparison unit 705 is used to compare the appearance features of the package to be associated in the image to be detected with the feature library of candidate associated packages if the movement speed of the package to be associated is less than a preset threshold, and determine the appearance feature similarity. The feature library includes the appearance features of candidate associated packages under multiple preset viewpoints. The association score calculation unit 706 is used to fuse motion feature similarity and appearance feature similarity to obtain the first association score between the package to be associated and the candidate associated packages. Package association unit 707: If the first association score is not less than a preset threshold, the package to be associated is associated with the candidate associated package.

[0100] It should be noted that the package association device provided in this application embodiment is used to implement the package association method provided in the above embodiment, and the specific implementation process corresponds to the package association method in the above embodiment. It can be referred to the aforementioned package association method, and will not be repeated here.

[0101] This application also provides an electronic device (i.e., a computer device), which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement any of the package association methods described in the above embodiments. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0102] Reference Figure 8 , Figure 8 This illustration shows the hardware structure of an electronic device according to another embodiment, the electronic device comprising: The processor 801 can be implemented using a general-purpose central processing unit (CPU), graphics processing unit (GPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the package association method of the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 505.

[0103] This application also provides a computer-readable storage medium storing a computer program for causing a computer to execute the package association method described in the above embodiments.

[0104] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0105] This invention also provides a computer program product that stores program instructions, which, when executed by a computer, cause the computer to implement the package association method described in any of the above embodiments.

[0106] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0107] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0110] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0111] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0113] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A package association method, characterized by, The method is applied to package tracking within a single camera, and the method includes: Acquire the image to be detected captured by the camera, and perform package detection on the image to be detected to determine at least one package to be associated in the image to be detected; For each package to be associated, candidate associated packages corresponding to the package to be associated are determined from the tracked packages contained in the previous frame image captured by the camera corresponding to the image to be detected. The predicted position is obtained by predicting the position of the candidate associated wrapper in the image to be detected, and the motion feature similarity is determined based on the positional deviation between the actual detected position of the candidate associated wrapper in the image to be detected and the predicted position. Perform a roll detection on the package to be associated. If the roll detection result indicates that the package to be associated has rolled, obtain the movement speed of the package to be associated in the image to be detected. If the movement speed of the package to be associated is less than a preset threshold, the appearance features of the package to be associated in the image to be detected are compared with the feature library of the candidate associated packages to determine the appearance feature similarity. The feature library includes the appearance features of the candidate associated packages under multiple preset viewpoints. The motion feature similarity and the appearance feature similarity are fused to obtain the first association score between the package to be associated and the candidate associated package; If the first association score is not less than a preset threshold, the package to be associated is associated with the candidate associated package.

2. The method of claim 1, wherein, The method for constructing the feature library includes: Obtain the 3D image of the candidate associated package during the main line acquisition phase, wherein the 3D image includes point cloud data and texture mapping data of the five faces of the candidate associated package excluding the bottom face; The point cloud data and texture mapping data of each face in the 3D image are rendered at multiple preset pose angles to obtain the rendered image and rendered point cloud of the candidate association wrapped in multiple viewpoints. Feature extraction is performed on each of the rendered images to obtain texture features under the corresponding viewpoint, and feature extraction is performed on each of the rendered point clouds to obtain geometric features under the corresponding viewpoint; The feature library of the candidate associated packages is constructed by associating and storing each viewpoint and the corresponding texture and geometric features.

3. The method of claim 2, wherein, Appearance features include texture features and geometric features. The step of comparing the appearance features to be associated and wrapped in the image to be detected with the feature library of candidate associated wrappers to determine appearance feature similarity includes: Obtain the historical pose set of the candidate associated package, the historical pose set including the viewpoints selected from the feature library corresponding to the candidate associated package based on the appearance features of the candidate associated package in the previous frame image or previous multiple frames image; If the distribution of each viewpoint in the historical pose set is the target distribution, the texture features and geometric features to be associated and wrapped are compared with the texture features and geometric features corresponding to each viewpoint in the historical pose set to obtain the first appearance matching score of each viewpoint. Remove viewpoints from the historical pose set whose first appearance matching score is less than a preset threshold. If the historical pose set is not empty after removal, determine the maximum score in the historical pose set after removal as the appearance feature similarity.

4. The method of claim 3, wherein, The step of comparing the appearance features of the package to be associated in the image to be detected with the feature library of the candidate associated packages to determine the appearance feature similarity further includes: If the historical pose set is empty after removal, the geometric features to be associated wrapped in the image to be detected are compared with the geometric features of the candidate associated wrapped in the previous frame image to obtain the appearance feature similarity.

5. The method of claim 3, wherein, The step of comparing the appearance features of the package to be associated in the image to be detected with the feature library of the candidate associated packages to determine the appearance feature similarity further includes: If the distribution of each viewpoint in the historical pose set is a non-target distribution, the appearance features of the package to be associated are compared with the texture features and geometric features corresponding to each viewpoint in the feature library corresponding to the candidate associated package to obtain the second appearance matching score of each viewpoint. Each viewpoint whose second appearance matching score is greater than a preset threshold is constructed as the current pose set, and the maximum score in the current pose set is determined as the appearance feature similarity.

6. The method of claim 4, wherein, The method further includes: If the historical pose set is empty after removal, mark the package to be associated as having its bottom face out. The state detection of the package to be associated is performed for consecutive frames of forward tracing to determine whether the feature update conditions are met. The feature update conditions include: the package to be associated is marked as bottom-facing for multiple consecutive frames, and the package to be associated is not in a tumbling transition state in the image to be detected. If the package to be associated satisfies the feature update condition, extract the bottom point cloud data of the package to be associated in the image to be detected; The bottom geometric features of the package to be associated are determined based on the bottom point cloud data, and the bottom geometric features are fused with the three-dimensional image of the candidate associated package to obtain an updated three-dimensional image. The updated 3D image is rendered at multiple pose angles to obtain a rendered image and a rendered point cloud from the new viewpoint. New texture features are obtained by extracting features from the rendered image from the new viewpoint, and new geometric features are obtained by extracting features from the rendered point cloud from the new viewpoint. The newly added viewpoint, the newly added texture features, and the newly added geometric features are associated and stored in the feature library of the candidate associated package.

7. The method of claim 1, wherein, The rolling detection of the package to be associated includes: Based on the appearance features of the package to be associated in the image to be detected and the appearance features of the candidate associated package in the previous frame image, determine the appearance change feature value; Based on the image to be detected and the previous frame image, determine the point cloud shape change or detection box shape change of the package to be associated; If the positional deviation is less than a preset threshold, the appearance change feature value is less than a preset threshold, and the point cloud shape change or the detection box shape change is greater than a preset threshold, a tumbling detection result indicating that the package to be associated has tumbled is generated.

8. The method of claim 1, wherein, The method further includes: If the movement speed of the package to be associated is not less than a preset threshold, a second association score between the package to be associated and the candidate associated package is determined based on the similarity of the movement features. If the second association score is not less than a preset threshold, the package to be associated is associated with the candidate associated package.

9. An electronic device, comprising: The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images