Probabilistic driven rgb-d semantic instance deduplication and inventorying method and system

Abstract

The present application belongs to the technical field of mobile robot vision perception and multi-target tracking, and particularly relates to a probability-driven RGBD semantic instance deduplication and inventory method and system, which comprises the following steps: obtaining a two-dimensional detection box, a category label and a three-dimensional observation vector of the inventory material in the RGB-D image; constructing a heteroscedastic observation uncertainty model related to the measurement distance, and dynamically assigning a diagonal covariance matrix to each observation; maintaining a confirmed instance state set, and matching the current observation with the existing instance based on the semantic hard constraint and the Mahalanobis distance soft constraint, and the correlation threshold is adaptively adjusted according to the covariance; the matched observation is updated to the three-dimensional position and state covariance of the instance through Kalman filtering, and the unmatched observation is registered as a new instance and included in the inventory list after being continuously confirmed by the life cycle state machine. The present application adaptively suppresses the depth noise through the probability-driven mechanism, effectively suppresses the repeated counting and identity switching, and improves the inventory accuracy and robustness.

Description

Technical Field

[0001] This invention belongs to the field of mobile robot visual perception and multi-target tracking technology, and particularly relates to probabilistic-driven RGB... A method and system for deduplication and inventory of semantic instances. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In automated inventory and asset verification tasks, mobile robots need to autonomously complete accurate statistics on the types and quantities of materials. Currently, deep learning-based object detection networks can provide the location and category information of objects in a single frame image. However, under continuous video stream observation, the same material instance will be detected repeatedly, and directly accumulating the results of a single frame will lead to a serious problem of duplicate counting.

[0004] Existing cross-frame target association and deduplication methods mainly fall into two categories. The first category is based on multi-target tracking algorithms on a two-dimensional image plane, using appearance features and motion prediction to maintain identity consistency. However, when objects at different depths overlap in projection on the image plane, appearance and two-dimensional motion features struggle to distinguish depth differences, leading to frequent identity switching. The second category utilizes RGB-D sensors to upscale the detection results to three-dimensional space, performing clustering association based on a fixed Euclidean distance threshold. However, consumer-grade depth sensors exhibit significant heteroscedasticity in their measurement noise, with the error standard deviation amplifying nonlinearly with measurement distance. Using a uniform threshold, nearby targets are prone to incorrect fusion, while distant targets may be misjudged as new instances due to depth fluctuations. Furthermore, in lightweight systems lacking high-precision global odometry compensation, local coordinate drift caused by camera motion further exacerbates these issues.

[0005] Therefore, there is an urgent need for a robust inventory method that can adapt to the noise distribution of depth sensors and effectively suppress duplicate counting in the absence of global pose correction. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention provides a probability-driven RGB The method and system for deduplication and inventory of D semantic instances aims to solve the problems of duplicate counting and identity switching caused by heteroscedastic noise of depth sensors and lack of global pose correction in the inventory of materials of mobile robots.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: Firstly, a probability-driven RGB was disclosed. Methods for deduplication and inventory of D semantic instances include: Obtain the two-dimensional detection bounding box, category label, and three-dimensional observation vector of the materials to be inventoried in the RGB-D image; Construct a heteroscedastic observation uncertainty model related to measurement distance, and dynamically assign a diagonal covariance matrix to each three-dimensional observation vector; A state set of confirmed material instances is constructed, and the current observation is associated with existing instances based on a spatial-semantic concatenation matching strategy. Semantic category consistency is used as a hard constraint, and Mahalanobis distance is used as a soft constraint. The association threshold is adaptively adjusted according to the covariance matrix. For observations that are successfully matched, the 3D position and state covariance of the instance are updated using Kalman filtering; for observations that are not matched, they are registered as new instances and included in the inventory list after continuous confirmation by the lifecycle state machine.

[0008] Furthermore, the process of obtaining the three-dimensional observation vector includes: constructing a region of interest in the central region of the two-dimensional detection box, thresholding the depth map, and taking the median of the effective depth values ​​as the representative depth.

[0009] Furthermore, based on the pinhole camera model, the center pixel coordinates of the detection box and the representative depth are back-projected onto the camera coordinate system to obtain the three-dimensional observation vector.

[0010] Furthermore, in the heteroscedasticity observation uncertainty model, the lateral variance of the diagonal covariance matrix is ​​set as a fixed constant; the depth direction variance is modeled as the sum of the standard deviation of the basic noise and the amplification factor of the squared distance.

[0011] Furthermore, the squared Mahalanobis distance follows a chi-square distribution with 3 degrees of freedom, and the association threshold is taken as the theoretical quantile corresponding to the significance level.

[0012] Furthermore, the Kalman filter update omits the kinematic prediction equation, calculates the Kalman gain using the current observations, and fuses the observation information to update the three-dimensional position mean and state covariance of the instance.

[0013] Furthermore, in the lifecycle state machine, newly created instances are initially in a transient state, and are only updated to a confirmed state and included in the inventory list after a preset number of consecutive successful matches.

[0014] Secondly, publicly probabilistic RGB A semantic instance deduplication and inventory system, including: The observation extraction module is used to acquire the two-dimensional detection box, category label and three-dimensional observation vector of the materials to be inventoried in the RGB-D image; The uncertainty modeling module is used to construct a heteroscedastic observation uncertainty model related to the measurement distance, and dynamically assign a diagonal covariance matrix to each three-dimensional observation vector; The cascaded matching module is used to maintain the state set of confirmed material instances and associate the current observation with existing instances based on the spatial-semantic cascaded matching strategy. In the cascaded matching strategy, semantic category consistency is used as a hard constraint, Mahalanobis distance is used as a soft constraint, and the association threshold is adaptively adjusted according to the covariance matrix. The state management module is used to update the 3D position and state covariance of the corresponding instance for successfully matched observations using Kalman filtering; for unmatched observations, they are registered as new instances and included in the inventory list after continuous confirmation by the lifecycle state machine.

[0015] Thirdly, a computer device is disclosed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the steps of the method described above.

[0016] Fourthly, a computer-readable storage medium is disclosed having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0017] The above one or more technical solutions have the following beneficial effects: This invention leverages the physical characteristic of nonlinear amplification of measurement noise with distance in RGB-D depth sensors, using Mahalanobis distance as a spatial correlation metric to adaptively change the tolerance of spatial matching with observation distance. Compared to traditional methods with fixed Euclidean distance thresholds, this invention fundamentally overcomes the contradictions of near-range targets being easily erroneously fused and far-range depth fluctuations being misjudged as new instances, significantly improving the accuracy and robustness of cross-frame correlation in complex depth scenes.

[0018] This invention cascades semantic category consistency as a hard constraint with Mahalanobis distance as a soft constraint, effectively distinguishing between similar and dissimilar targets at different depths in complex scenarios such as projection overlap. This avoids the problem of frequent identity switching in occluded or overlapping scenarios in multi-target tracking algorithms based on two-dimensional appearance features, ensuring the correctness and identity consistency of material category attribution in the inventory list.

[0019] This invention effectively suppresses false instances introduced by single-frame visual misdetection through a triggering mechanism that requires confirmation after multiple consecutive frames of matching. Simultaneously, based on the reasonable assumption that materials are stationary, it eliminates the timeout forgetting mechanism, preventing the erroneous removal of confirmed instances due to brief occlusion. These two mechanisms work synergistically to ensure the accuracy and completeness of inventory results.

[0020] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a probability-driven RGB model according to Embodiment 1 of the present invention. Flowchart of D semantic instance deduplication and inventory method; Figure 2 This is a flowchart of the heteroscedasticity observation uncertainty model according to Embodiment 1 of the present invention; Figure 3 This is a flowchart illustrating the cascaded matching and lifecycle state machine fusion process of Embodiment 1 of the present invention. Figure 4 This is a schematic diagram of the heteroscedasticity fitting results of depth error standard deviation and distance in Embodiment 1 of the present invention; Figure 5 This is a diagram showing the results of material identification and inventory in Embodiment 1 of the present invention. Detailed Implementation

[0023] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0025] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0026] Example 1 like Figure 1 As shown, this embodiment discloses a probability-driven RGB Methods for deduplication and inventory of D semantic instances include: Step 1: Obtain the two-dimensional detection box, category label, and three-dimensional observation vector of the materials to be inventoried in the RGB-D image; At any time First, an object detection network is used to perform real-time inference on the color images captured by the RGB-D camera. In this embodiment, the YOLOv8 object detection network is used as an example implementation, but those skilled in the art should understand that other types of deep learning object detectors can also be used.

[0027] Specifically, the RGB image is input into the YOLOv8 network, which outputs the two-dimensional bounding boxes of all detected targets in the frame and their corresponding category labels. Let the top-left pixel coordinates of a single target bounding box be... The bottom right pixel coordinates are The center pixel coordinates of the bounding box are : Center pixel Construct a rectangular region of interest (ROI) for the geometric center. The width and height of the ROI are half the width and height of the corresponding bounding box, respectively, to ensure that the region covers the target subject and avoids interference from the edge background.

[0028] Perform a threshold truncation operation on depth pixels within the ROI: preset a minimum depth threshold based on the effective range of the RGB-D sensor. With maximum depth threshold Remove depth values ​​less than or greater than Invalid pixels; sort the remaining valid depth values ​​in ascending order and take the median as the representative depth d of the target to suppress the influence of depth outliers and local noise.

[0029] Then, based on the pinhole camera model, the center coordinates of the bounding box are... With representative depth By combining these methods, the three-dimensional observation vector of the target in the camera coordinate system can be obtained through back projection. :

[0030] in , , , These are camera intrinsic parameters, obtained through prior camera calibration. , pixel coordinate system , The equivalent focal length in a direction is calculated by converting the camera's physical focal length into the physical size of a single pixel; , The camera optical center in pixel coordinate system , The principal point offset of the direction, i.e., the pixel coordinates of the intersection point of the optical axis and the image plane.

[0031] Step 2: Construct a heteroscedastic observation uncertainty model related to the measurement distance, and dynamically assign a diagonal covariance matrix to each three-dimensional observation vector; Because the measurement noise of RGB-D cameras has significant heteroscedasticity, the standard deviation of its depth random error is nonlinearly amplified with the increase of measurement distance, so it is impossible to use a single fixed covariance matrix to uniformly characterize all observations.

[0032] Based on this, this embodiment constructs a heteroscedasticity observation uncertainty model related to the measurement distance, specifically: Observing the covariance matrix Taken as a diagonal matrix:

[0033] Among them, the horizontal variance Set as a fixed constant to absorb the positioning jitter of the 2D detection box on the image plane and the local odometry drift caused by the lack of global pose correction in lightweight systems; depth direction variance. Modeling as a depth-dependent representation of the target The square function:

[0034] in This refers to the depth represented by the target extracted in step 1. The base depth noise standard deviation reflects the inherent measurement fluctuations of the sensor at close range. is the distance attenuation amplification factor, which characterizes the rate at which depth error increases with distance.

[0035] In this embodiment, and The depth images were obtained through the following offline sensor calibration method: An RGB-D camera was fixed at a series of known distances, and multiple frames of depth images were continuously acquired for the same stationary planar target at each distance; for each distance, the representative depth of the target was extracted and back-projected to obtain the three-dimensional coordinates, and the standard deviation of the inter-frame random fluctuation of the depth direction coordinates was calculated as the depth image at that distance. Value; used to measure distance For independent variable, corresponding As the dependent variable, the least squares method is used to fit a quadratic function. Thus determine and The specific value.

[0036] Step 3: Construct a state set of confirmed material instances, and associate the current observation with existing instances based on the spatial-semantic concatenation matching strategy. Semantic category consistency is used as a hard constraint, and Mahalanobis distance is used as a soft constraint. The association threshold is adaptively adjusted according to the covariance matrix. Maintain a set of confirmed material instances in the local coordinate system. Each instance Includes its three-dimensional geometric center prior mean State covariance matrix and category tags This collection records all individual materials that are currently being tracked stably.

[0037] For the new observation set With existing instance sets Perform cascading association matching, such as Figure 2 As shown. Among them, the new observation set It consists of all valid 3D observation vectors extracted in step 1 of the current frame. Each observation element contains the 3D spatial coordinates of the target and the corresponding semantic category label.

[0038] The first level implements hard semantic constraints for each observation category. With each instance category Class comparison is performed only when the observed class With instance category Only when there is strict consistency can the observation-instance pair be eligible for subsequent spatial distance assessment; if the categories are inconsistent, the pair is rejected outright, and no further spatial calculations are performed. This hard constraint ensures that different types of materials are not incorrectly associated, guaranteeing the accuracy of the inventory list at the semantic level.

[0039] The second level is the Mahalanobis distance soft constraint: For observation-instance pairs subject to semantic hard constraints, the squared Mahalanobis distance between them is calculated as follows: Define observation residuals Observation residual covariance Then the squared Mahalanobis distance is defined as:

[0040] In the formula: This is the current 3D observation vector (3×1 column vector) to be matched, corresponding to the spatial coordinates of a single material target in the camera coordinate system; This is the prior mean of the three-dimensional position of the corresponding tracked instance (3×1 column vector), which is the current best position estimate of the instance. The covariance matrix (3×3 diagonal matrix) of the observation vector is dynamically generated by the heteroscedasticity observation uncertainty model in step 2 based on the observation distance; The state covariance matrix (3×3 diagonal matrix) of the tracked instances represents the degree of uncertainty in the instance location estimation; The observation residual vector reflects the deviation between the current observation and the prior position of the instance; The observation residual covariance matrix is ​​composed of the superposition of observation uncertainty and instance state uncertainty.

[0041] Under the assumption that the observed noise follows a zero-mean Gaussian distribution, Obeying a system with 3 degrees of freedom (Chi-square) distribution. The degrees of freedom are 3, corresponding to the three independent coordinate dimensions of x, y, and z in three-dimensional space; the significance level is selected. The corresponding upper quantile of the chi-square distribution is used as the association threshold, significance level. This represents the upper limit of the statistical probability of "the same instance being incorrectly identified as different instances".

[0042] Preferably, in this embodiment, Corresponding theoretical threshold Among all candidate pairs that meet the association threshold, the observation-instance pair with the smallest Mahalanobis distance is selected as the matching result. Successfully matched observations update the state of their corresponding instances; unmatched observations are marked as new candidate instances.

[0043] Step 4: For successfully matched observations, update the 3D position and state covariance of the instance using Kalman filtering; For successfully matched observation-instance pairs, Kalman filtering measurement updates are performed. Current observation information is then fused into the state of the confirmed instances to achieve smooth convergence and uncertainty recursion of the material's 3D location.

[0044] Since the inventory items are stationary in the environment, and the instance positions do not substantially shift between adjacent frames when the processing frame rate is high, this embodiment omits the kinematic prediction equation and directly calculates the Kalman gain using the current observations. :

[0045] In the formula: The prior state covariance matrix (3×3 diagonal matrix) of the instance to be updated is inherited from the tracking result of the previous frame, representing the uncertainty of the current position estimation of the instance; The observation noise covariance matrix (3×3 diagonal matrix) is generated dynamically by the heteroscedasticity observation uncertainty model in step 2 based on the target measurement distance, and represents the noise level of this three-dimensional measurement.

[0046] By fusing observation residuals using Kalman gain, the 3D position mean and state covariance of the instance are updated:

[0047] In the formula: This is the posterior position mean (3×1 column vector) after fusion of observations, which is the optimal 3D position estimate of the instance after this round of updates; The mean of the prior positions before the update is a 3×1 column vector, which is the estimated 3D position of the instance inherited from the previous frame; The current frame's three-dimensional observation vector is a 3×1 column vector, obtained through back projection of the pinhole camera model in step 1, corresponding to the spatial coordinates of the materials measured in this instance. The posterior state covariance matrix after fusion of observations is a 3×3 diagonal matrix, representing the uncertainty of the updated instance location estimate; It is a 3x3 identity matrix with 1s on the main diagonal and 0s on the rest. It is used to match the dimension of matrix operations and ensure the validity of algebraic operations.

[0048] The main function of this fusion process is to improve the observation covariance allocated in step 2 when the observation distance is long. Auto-amplification, Kalman gain The corresponding reduction in distance automatically suppresses the correction weights for instance location estimation from long-distance observations, preventing excessive noise from causing location drift. Conversely, when the observation distance is close, the observation covariance is small, the Kalman gain increases, and high-confidence observations can effectively converge the three-dimensional coordinates of instances.

[0049] Therefore, this embodiment achieves Bayesian optimal estimation of the position of stationary materials without relying on global pose correction.

[0050] Step 5: For unmatched observations, after continuous confirmation by the lifecycle state machine, register them as new instances and include them in the inventory list.

[0051] To filter out instantaneous visual false detections, such as false detections in a single frame caused by sudden changes in lighting, reflections, or occluded edges, this embodiment designs a Hit-Confirmation lifecycle state machine to verify the stability of newly registered instances.

[0052] Specifically, unmatched observations in step 3 are marked as new candidate instances, and their initial state is set to Tentative. These candidate instances participate in cascaded matching in subsequent consecutive frames, and the system accumulates the number of successful matches. Matching is only considered successful when the instance has been successfully matched a preset number of times in consecutive frames. Then, in this embodiment, we take Its state transitions from Tentative to Confirmed, and it is officially added to the valid inventory list; if a matching failure occurs before the number of confirmations is reached, the counter is reset and the candidate instance is discarded.

[0053] Furthermore, based on the core assumption of static environmental materials, this embodiment omits the timeout forgetting mechanism, meaning that instances are not removed from the inventory list simply because they have not been observed for an extended period. This design effectively prevents missed inventory entries due to temporary obstruction, such as when the robot's perspective changes or when personnel temporarily pass by, ensuring that confirmed material instances are continuously retained once registered, thus guaranteeing the integrity of the inventory results.

[0054] In this embodiment, the lifecycle state machine adopts an acknowledgment triggering mechanism: new observations are initially recorded as provisional states, and only after a preset number of consecutive successful matches do they transition to the acknowledgment state and be included in the valid inventory list; for stationary materials, the forgetting mechanism is removed to avoid missing inventory due to long-term obstruction.

[0055] To verify the physical rationality of the heteroscedasticity observation uncertainty model established in this embodiment, the depth measurement noise of the RGB-D sensor was first calibrated offline. Figure 3 The heteroscedasticity fitting results between the standard deviation of depth error and the measured distance are presented, where the horizontal axis represents the measured distance (unit: m) and the vertical axis represents the standard deviation of the depth random error (unit: mm).

[0056] The calibration process is as follows: The camera is fixed at different distances, and multiple frames of depth data are continuously acquired for the same stationary object. The random fluctuations in its back-projected coordinates are then statistically analyzed. The discrete points on the curve in the figure represent the measured statistical results, and the solid line is the least-squares fitting curve based on the physical degradation model. Figure 3 As can be seen, the standard deviation of depth error increases significantly with the square of the distance. At 1.0m, the standard deviation is approximately 3mm, while at 3.5m it rises to approximately 26mm, with a goodness of fit R²≈0.996. This result conclusively verifies the heteroscedasticity observation uncertainty model established in this invention. Its physical rationality and engineering applicability.

[0057] To further illustrate the advantages of this implementation method, a comparative experiment was conducted on a self-built test sequence containing 300 consecutive observations. For example... Figure 4 This is a diagram showing the results of material identification and inventory in this embodiment.

[0058] in, Figure 4 The left side shows the real-time recognition effect of the YOLOv8 object detection network on materials in the scene during the mobile robot inspection process. Each material instance was accurately detected and attached with a two-dimensional bounding box and category label. Figure 4 The right side displays the final inventory statistics after processing by the probability-driven cascaded matching and lifecycle state machine of this invention. Figure 4 As can be seen, the method in this embodiment successfully confirmed all 6 actual material instances in the scenario correctly, with no duplicate counts, and the inventory accuracy rate reached 100%.

[0059] The experiment included six objects such as water bottles, keyboards, and books, positioned at different depths from 0.4 m to 1.1 m in front of the camera, with deliberate local camera translation and rotation jitter. Four strategies were compared: (a) no deduplication baseline; (b) Euclidean distance clustering (fixed threshold 0.2 m); (c) 2D multi-object tracking DeepSORT; and (d) the method of this invention.

[0060] Table 1 shows a comparison of the inventory counting performance of each method: Table 1. Performance Comparison of Different Inventory Deduplication Methods Serial Number Comparison Methods Inventory accuracy Number of identity switching (a) Baseline without deduplication Extremely low — (b) Euclidean distance clustering (0.2m) 60.0% 4 (c) 2D Multi-Object Tracking DeepSORT 75.0% 3 (d) Method of the present invention 100% 0 Note: Inventory accuracy is defined as the ratio of the number of correctly matched real instances to the total number of instances output by the system; the no-deduplication baseline scheme directly accumulates the detection results of a single frame, resulting in a large number of duplicate counts, and the accuracy is much lower than other schemes, so specific values ​​and the number of identity switching were not counted.

[0061] The results show that the accuracy of the inventory method in this embodiment (the ratio of the number of correctly matched real instances to the total number of detected feedback instances) reaches 100%, and the number of identity switching is 0; while the accuracy of Euclidean distance clustering is only 60.0%, with 4 identity switching, and the accuracy of DeepSORT is 75.0%, with 3 identity switching.

[0062] To verify the effectiveness of each core module of this invention, an ablation experiment was designed using the controlled variable method. The heteroscedasticity observation uncertainty model and the lifecycle state machine module were removed respectively, and the performance changes were compared under the same test sequence. The results are shown in Table 2. Table 2 Comparison of Ablation Experiment Performance of Core Modules Serial Number Algorithm Configuration Inventory accuracy Number of identity switching 1 Complete Algorithm (This Invention) 100% 0 2 Model for removing heteroscedasticity observation uncertainty 37.5% 10 3 Remove lifecycle state machine 85.7% — Ablation experiments further confirmed that if the heteroscedasticity model was removed and replaced with a fixed minimum covariance, the accuracy dropped to 37.5%, with up to 10 identity switching incidents, indicating that heteroscedasticity modeling is the core to ensuring the accuracy of cross-depth scene association. If the state machine was removed to directly confirm entry into the database, the accuracy dropped to 85.7%, indicating that the lifecycle state machine can effectively filter false detections in a single frame and improve the accuracy of inventory results. When generalizing on a 596-frame internationally published standard test sequence, the method in this embodiment still successfully locked 16 material instances without global pose compensation, fully verifying the robustness of the probabilistic framework in complex dynamic environments.

[0063] Example 2 Based on the method described in Embodiment 1, the purpose of this embodiment is to provide a probability-driven RGB A semantic instance deduplication and inventory system, including: The observation extraction module is used to acquire the two-dimensional detection box, category label and three-dimensional observation vector of the materials to be inventoried in the RGB-D image; The uncertainty modeling module is used to construct a heteroscedastic observation uncertainty model related to the measurement distance, and dynamically assign a diagonal covariance matrix to each three-dimensional observation vector; The cascaded matching module is used to maintain the state set of confirmed material instances and associate the current observation with existing instances based on the spatial-semantic cascaded matching strategy. In the cascaded matching strategy, semantic category consistency is used as a hard constraint, Mahalanobis distance is used as a soft constraint, and the association threshold is adaptively adjusted according to the covariance matrix. The state management module is used to update the 3D position and state covariance of the corresponding instance for successfully matched observations using Kalman filtering; for unmatched observations, they are registered as new instances and included in the inventory list after continuous confirmation by the lifecycle state machine.

[0064] Example 3 The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0065] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium.

[0066] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above-described method.

[0067] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0068] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0069] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A probability-driven RGB The method for deduplication and inventory of D semantic instances is characterized by... include: Obtain the two-dimensional detection bounding box, category label, and three-dimensional observation vector of the materials to be inventoried in the RGB-D image; Construct a heteroscedastic observation uncertainty model related to measurement distance, and dynamically assign a diagonal covariance matrix to each three-dimensional observation vector; A state set of confirmed material instances is constructed, and the current observation is associated with existing instances based on a spatial-semantic concatenation matching strategy. Semantic category consistency is used as a hard constraint, and Mahalanobis distance is used as a soft constraint. The association threshold is adaptively adjusted according to the covariance matrix. For observations that are successfully matched, the 3D position and state covariance of the instance are updated using Kalman filtering; for observations that are not matched, they are registered as new instances and included in the inventory list after continuous confirmation by the lifecycle state machine.

2. The probability-driven RGB as described in claim 1 The method for deduplication and inventory of D semantic instances is characterized by... The process of obtaining the three-dimensional observation vector includes: constructing a region of interest in the central region of the two-dimensional detection box, truncating the depth map with a threshold, and taking the median of the effective depth values ​​as the representative depth.

3. The probability-driven RGB as described in claim 2 The method for deduplication and inventory of D semantic instances is characterized by... Based on the pinhole camera model, the coordinates of the center pixel of the detection box and the representative depth are back-projected onto the camera coordinate system to obtain the three-dimensional observation vector.

4. The probability-driven RGB as described in claim 1 The method for deduplication and inventory of D semantic instances is characterized by... In the heteroscedasticity observation uncertainty model, the lateral variance of the diagonal covariance matrix is ​​set as a fixed constant; the depth direction variance is modeled as the sum of the standard deviation of the basic noise and the amplification factor of the squared distance.

5. The probability-driven RGB as described in claim 1 The method for deduplication and inventory of D semantic instances is characterized by... The squared Mahalanobis distance follows a chi-square distribution with 3 degrees of freedom, and the association threshold is taken as the theoretical quantile corresponding to the significance level.

6. The probability-driven RGB as described in claim 1 The method for deduplication and inventory of D semantic instances is characterized by... The Kalman filter update omits the kinematic prediction equation, calculates the Kalman gain using the current observations, and fuses the observation information to update the three-dimensional position mean and state covariance of the instance.

7. The probability-driven RGB as described in claim 1 The method for deduplication and inventory of D semantic instances is characterized by... In the lifecycle state machine, a new instance is initially in a transient state, and is only updated to a confirmed state and included in the inventory list after a preset number of consecutive successful matches.

8. Probability-driven RGB A semantic instance deduplication and inventory system for D, characterized in that... include: The observation extraction module is used to acquire the two-dimensional detection box, category label and three-dimensional observation vector of the materials to be inventoried in the RGB-D image; The uncertainty modeling module is used to construct a heteroscedastic observation uncertainty model related to the measurement distance, and dynamically assign a diagonal covariance matrix to each three-dimensional observation vector; The cascaded matching module is used to maintain the state set of confirmed material instances and associate the current observation with existing instances based on the spatial-semantic cascaded matching strategy. In the cascaded matching strategy, semantic category consistency is used as a hard constraint, Mahalanobis distance is used as a soft constraint, and the association threshold is adaptively adjusted according to the covariance matrix. The state management module is used to update the 3D position and state covariance of the corresponding instance for successfully matched observations using Kalman filtering; for unmatched observations, they are registered as new instances and included in the inventory list after continuous confirmation by the lifecycle state machine.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the probability-driven RGB according to any one of claims 1-7. The steps of deduplication and inventory methods for D semantic instances.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program performs the probability-driven RGB as described in any one of claims 1-7 above. The steps of deduplication and inventory methods for D semantic instances.

Images