A method for object-level semantic mapping and workstation alignment, and its storage medium.

Abstract

This invention discloses an object-level semantic mapping and workstation alignment method and storage medium, comprising: performing open-domain target detection on selected keyframes based on obtained RGB-D data and spatial pose to obtain corresponding detection boxes, and performing instance segmentation on the detection boxes to obtain pixel-level masks; adaptively weighting and fusing the three CLIP visual features according to the mask pixel area ratio to obtain the object-level semantic embedding corresponding to the mask; updating the objects in the global object map based on the established object-level point cloud to obtain object-level mapping, and uniformly transforming the calibration grid points in each workstation coordinate system to the robot navigation map coordinate system. This invention adapts to targets of different scales by adaptively fusing the three CLIP features according to the mask area, avoiding feature effect decay, and achieves unified alignment of workstation coordinates by relying on image feature matching, so that the map alone can directly support the actual operation of the robot.

Description

Technical Field

[0001] This invention relates to the field of robot environmental perception technology, and in particular to an object-level semantic mapping and workstation alignment method and storage medium. Background Technology

[0002] The characteristics that instruments in a chemical laboratory typically have can include:

[0003] (1) There are a large number of equipment of the same model, which are similar in appearance but located in different work positions;

[0004] (2) Some instruments are transparent, mirrored or have no texture in some areas, making it difficult to achieve stable correlation based solely on color or geometric information;

[0005] (3) The space between the walkway and the operating table is limited, and the robot's observable angle of view of the same instrument is narrow;

[0006] (4) Subsequent tasks require that objects have both spatial location and stable semantic embedding in the map to support open vocabulary query and cross-task reuse.

[0007] To identify various objects in a chemical laboratory setting and assign them semantic information, it is necessary to achieve object-level semantic mapping in the chemical laboratory setting. This will provide a standard spatial model and unified coordinate basis for laboratory equipment positioning, intelligent scheduling, autonomous operation, and environmental management, enabling the intelligent and efficient operation of the laboratory.

[0008] Existing object-level semantic mapping solutions mostly employ offline methods to process pre-recorded videos, such as ConceptGraphs class methods. However, these solutions have the following shortcomings:

[0009] (1) The cross-frame association stage usually relies on only a single geometric overlap or a single semantic similarity, which can easily lead to misassociations between instruments of the same model;

[0010] (2) The multi-view feature aggregation stage often uses arithmetic average, which lacks the ability to suppress occasional erroneous observations;

[0011] (3) The object map maintenance phase lacks a hard upper bound constraint on the point cloud bounding box. After long-term operation, the point cloud of a single object will slowly expand with the number of observations.

[0012] (4) During the pose consistency maintenance phase, there is a cumulative drift between the existing object point cloud and the latest optimized pose;

[0013] In view of this, this invention is hereby proposed. Summary of the Invention

[0014] The purpose of this invention is to provide a method and storage medium for object-level semantic mapping and workstation alignment, so as to reduce mismatches of similar experimental instruments and avoid feature effect decay, thereby solving the technical problems existing in the prior art.

[0015] The objective of this invention is achieved through the following technical solution:

[0016] A method for aligning object-level semantic mapping with workstations includes:

[0017] Receive color image data (RGB-D data) from the robot's depth camera, and call the Visual Simultaneous Localization and Mapping (SLAM) backend to perform pose estimation, obtaining the corresponding RGB-D data and spatial pose;

[0018] Based on the obtained RGB-D data and spatial pose, open domain target detection is performed on the selected keyframes to obtain the corresponding detection boxes, and the detection boxes are segmented into instances to obtain pixel-level masks.

[0019] For each of the pixel-level masks, mask cropping and mask masking are extracted in parallel, and these are combined with the full image of the key frame to form the CLIP visual features of the three-way image pre-training model. The CLIP visual features of the three-way image pre-training model are adaptively weighted and fused according to the proportion of mask pixel area, so as to obtain the object-level semantic embedding of the observation corresponding to the mask.

[0020] By utilizing depth maps and in-camera participation in camera pose, the effective pixels within each mask are back-projected from two dimensions to three dimensions to the world coordinate system to establish an object-level point cloud.

[0021] The object-level point cloud is assigned to the correspondence formed by the previous frame's 3D projection. A joint score of geometric similarity and semantic similarity is constructed for the remaining observations and a greedy allocation is performed. After the object update in the global object map is completed, the object-level map is obtained.

[0022] Based on the object-level mapping, the local Gaussian reconstructed scene of each workstation and the selected keyframe are matched with sparse and dense features under mask constraints to construct the correspondence between two-dimensional and two-dimensional and two-dimensional and three-dimensional; and the calibration grid points in the coordinate system of each workstation are uniformly transformed to the robot navigation map coordinate system.

[0023] The method further includes: point cloud cleaning of the object-level point cloud, and maintenance of a multi-view semantic feature library and closed-loop processing of 3D projection feedback; wherein,

[0024] The point cloud cleaning process includes:

[0025] Voxel downsampling is performed on the back-projected point cloud, and noise and artifacts are removed based on neighborhood distance.

[0026] The maintenance process of the multi-perspective semantic feature library includes:

[0027] For each object, a feature library is maintained that preserves several viewpoints, and the aggregated semantic embedding of the object is obtained by using a weighted average based on cosine distance based on this feature library.

[0028] The three-dimensional projection feedback closed-loop processing process includes:

[0029] Once the total number of keyframes exceeds the warm-up threshold, for each 3D point in all confirmed objects, the camera coordinates are first obtained by transforming the world position of the current frame to the camera pose and then to the camera coordinate system. for:

[0030] ;

[0031] in, For rotation matrix, For world point coordinates, It is a translation vector;

[0032] Then, the pixel coordinates are obtained through perspective projection. Projected depth of that point :

[0033] ;

[0034] in, , The focal lengths of the camera in the x and y directions are... , , Let the coordinates of a 3D point be the three axes in the camera coordinate system. , The pixel offset coordinates of the principal point of the image;

[0035] Generate a projection mask and bounding box based on all projected pixels, and count the number of projected points within the image. The number of points with consistent depth is Then the visibility ratio for:

[0036] ;

[0037] Only when And the projection mask pixels and At that time, the projection mask forms a valid projection cue box, and the corresponding projection confidence level... for:

[0038] ;

[0039] in, The median depth of the projection point. The maximum depth of the projection point;

[0040] The projection prompt box is fused with the target detection box of the next frame and serves as a unified prompt input for the instance segmentation model, thus forming a closed loop of detection-projection-confirmation.

[0041] This method also includes a periodic global merging and fragment timeout cleanup process, wherein:

[0042] The process of periodic global merging includes:

[0043] Every preset number of keyframes, the object map is traversed. Based on the dual threshold judgment rule of 3D point cloud overlap and aggregated feature cosine similarity, global merging is performed on object pairs that are geometrically overlapping, semantically consistent, of the same category and with adjacent centroids. Furthermore, for the robust baseline after global merging, the larger value of the two merged baselines is selected.

[0044] The fragment timeout cleanup process includes:

[0045] Periodically clean up objects that have not been matched for a long time and have been observed less than the predetermined number.

[0046] The method also includes a final state merging and prototype verification process after the session ends, wherein:

[0047] The final state merging process after the session ends includes:

[0048] The final state merging is performed iteratively using a relaxed threshold until no new merges are generated or the maximum number of iterations is reached. The relaxed threshold is a threshold that is greater than a predetermined amount of the standard threshold. The relaxed threshold includes: a relaxed threshold for merging the centroid distance of the final state, a relaxed threshold for merging the three-dimensional overlap of the final state, and a relaxed threshold for merging the semantic similarity of the final state.

[0049] The prototype verification process includes:

[0050] Assume offline training set for each object category Constructed category prototype for:

[0051] ;

[0052] in, For L2 norm normalization, For the set of all training samples corresponding to category c, This refers to the image and its corresponding detection box sample. For image feature extraction functions, For image cropping operations;

[0053] For each object Calculate prototype similarity With projection ratio for:

[0054] ;

[0055] in, For cosine similarity, The aggregated feature obtained after weighted fusion of the j-th object. For objects Category For objects Category The category prototype feature vector, The number of observations obtained from projection matching. The number of observations directly detected by the detector;

[0056] object Given the confirmed information, determine whether the object is abnormal or suspicious. The method for determining it is as follows:

[0057] ;

[0058] in, For objects The status has been confirmed. The threshold for determining prototype similarity. This is the threshold for determining the projection ratio; objects deemed suspicious are marked or removed during export. For logical AND, For logical OR.

[0059] The pose estimation is implemented based on a three-level pose correction mechanism, and includes:

[0060] The first level monitors the large change index counter exposed by the SLAM system. This counter increments after loop closure fusion or global bundle adjustment. When it changes, a correction transformation derived from the pose change of the corresponding contributing keyframe is applied to all object point clouds, and a global merging is triggered.

[0061] The second level involves iterating through the contribution keyframes of all objects every preset number of keyframes, updating and correcting objects whose correction amount is greater than the periodic correction threshold, so that the correction corresponding to the difference between their old and new poses is applied to the point cloud of that object.

[0062] The third level involves updating and correcting objects whose periodic correction amount is greater than the full correction threshold before the end of the mapping and observation process, whereby the periodic correction threshold is greater than the full correction threshold.

[0063] And the update correction includes: for each object Maintain its contribution keyframe set And apply the following formula to its point cloud:

[0064]

[0065] ;

[0066] in, This represents the increment of the K-pose change in the keyframe. To optimize the pose transformation matrix of the previous keyframe K, To optimize the latest pose matrix of keyframe K, The average rotation correction across all associated keyframes is used as the overall average rotation correction value for the object. This is an orthogonalization process based on singular value decomposition. for The split rotation increment, for Decompose the translation increment, The average translation correction across all associated keyframes is used as the overall translation correction value. To align the point coordinates with the latest pose after correcting the point cloud coordinates. The 3D point coordinates of the object before point cloud coordinate correction. For objects The set of keyframes that contribute to the project.

[0067] The process of updating objects in the global object map includes:

[0068] The projection pre-matching stage, which assigns the object-level point cloud to the corresponding relationship formed by the previous frame's 3D projection, includes:

[0069] In the correspondence between the detection box and the projection box, if the Intersection over Union (IoU) of the two is greater than the threshold value of the IoU between the two-dimensional detection box and the projection box... Then, the observation-object correspondence is directly established, and then semantic verification is performed. This was subsequently confirmed; among them, The cosine similarity between the currently observed features and the aggregated features of the object. The threshold for semantic similarity determination. Let i be the final fused semantic features of the single target in frame t. The aggregate feature of the j-th object in the map;

[0070] The global greedy search phase, which constructs a joint score of geometric and semantic similarity for the remaining observations and performs a greedy assignment, includes:

[0071] For the remaining unmatched observations With unoccupied candidate objects The similarity is defined as follows:

[0072] 3D point cloud overlap for:

[0073] ;

[0074] in, For the current observation target point cloud, For the current observation of three-dimensional points, For 3D points of objects on a map, A collection of three-dimensional point clouds of objects. The threshold for spatial distance determination. To obtain the minimum value;

[0075] The IoU of a 2D projection mask is The calculation method is as follows:

[0076] ;

[0077] in, For the target mask in the current frame, A two-dimensional mask obtained by backprojecting map objects;

[0078] Geometric similarity With semantic similarity They are respectively:

[0079] ;

[0080] ;

[0081] Joint rating for:

[0082] ;

[0083] in, These are the weighting coefficients;

[0084] according to Greedy selection in descending order exceeds the threshold for effective matching. The correspondence is such that each observation is matched with each object at most once;

[0085] It also includes performing a dual anti-inflation gating process, namely:

[0086] After each successful match, the current observed point cloud is used. Update existing objects on the map Data If so, the following two checks will be performed:

[0087] Inspection based on centroid drift gating:

[0088] ;

[0089] in, This represents the Euclidean offset distance between the centroids of the currently observed object and existing objects on the map. The current observation point cloud centroid, Set the centroid of the point cloud for the existing objects on the map. This is the allowable threshold for positional offset; based on this formula, the offset of the center of the old and new objects can be determined to avoid unreasonable drift of the object's position.

[0090] Inspection based on absolute volume gating:

[0091] ;

[0092] ;

[0093] in, This represents the diagonal length of the 3D bounding box of the object after merging the old and new point clouds, used to characterize the overall spatial dimensions of the merged object. The overall point cloud after merging the old and new point clouds. This is the upper limit of the size magnification. The median length of the diagonal of the bounding box observed for this object;

[0094] If any gate fails, the object update is rejected; if rejected and the observation confidence is higher than the predetermined value, it is created as a new object; otherwise, it is discarded.

[0095] After constructing the correspondences from two dimensions to two dimensions and from two dimensions to three dimensions, the process also includes determining the seven-degree-of-freedom similarity transformation, and this process includes:

[0096] Seven-degree-of-freedom similarity transformation Estimate the objective function for:

[0097] ;

[0098] in, For the index of the matching point pair, These are respectively the scale degrees of freedom, the three-dimensional rotation matrix, and the three-dimensional translation vector. For the Gaussian side three-dimensional matching point, For SLAM side 3D matching points;

[0099] The closed-form solutions of the Random Sampling Consensus Algorithm (RANSAC) and the Umeyama algorithm are given. Initial value; when the physical spacing of the calibration grid is known. At that time, the scale degrees of freedom Locked as: ,in, This represents the average distance between grid points in the local coordinate system.

[0100] Regarding the The initial value is refined using the Levenberg-Marquardt nonlinear algorithm, and parameterized as follows: ,in, For logarithmic mapping, This is the transformation from a rotation matrix to a rotation vector. It is a 7-dimensional real number space;

[0101] The ICP registration iterative process is executed to minimize the distance error between corresponding points in the two point clouds, and the rotation matrix R and translation vector t are updated iteratively and synchronously; and the sampling radius of the dense point cloud of ICP on the SLAM side is... The physical bounding box diagonal of the Gaussian point cloud is given as follows:

[0102] ;

[0103] in, , Let be the diagonal length of the physical bounding box of the Gaussian point cloud. Let the physical scale coordinates of the Gaussian point cloud be... To calibrate the scale of the grid, These are the coordinates in the local coordinate system of the Gaussian model.

[0104] by cv2.solvePnPRansac Seven-degree-of-freedom similarity transformation Perform independent cross-validation, where... cv2.solvePnPRansac This is a solution for PnP based on RANSAC in OpenCV, where PnP is a perspective problem with n points.

[0105] Disambiguation of multiple instances corresponding to the same category of workstations, and construction of quality scores. for:

[0106] ;

[0107] in, The number of interior points in RANSAC. For interior point proportions, For scale ratio;

[0108] It performs two rounds of greedy allocation: the first round strictly allocates object instances to workstation scenes at a 1:1 ratio; the second round, when the number of object instances exceeds the number of workstations, allows the remaining objects to reuse already allocated workstation scenes of the same model, with the allocation key set to... To maintain uniqueness, among which, This is the name of the workstation scene. Number the object instances; select the final optimal R and t from multiple candidate results to determine the final seven-degree-of-freedom similarity transformation. ;

[0109] The process of uniformly transforming the calibration grid points in each workstation coordinate system to the robot navigation map coordinate system includes:

[0110] Based on the correspondence between the laser poses of robot navigation waypoints and the corresponding SLAM camera poses, a least-squares fitting two-dimensional similarity transformation is performed:

[0111]

[0112] in, The coordinates are in the robot's navigation map coordinate system. The scale factor for SLAM to map. It is a two-dimensional rotation matrix. For rotation angle, The horizontal coordinates in the SLAM coordinate system It is a two-dimensional translation vector;

[0113] The least squares fitting two-dimensional similarity transformation is compared with the final seven-degree-of-freedom similarity transformation. The coordinates of each workstation are combined and the calibration grid points in each workstation coordinate system are uniformly transformed to the robot navigation map coordinate system.

[0114] The process of forming the visual features of the three-way image pre-training model CLIP involves processing each effective mask... Parallel extraction of three CLIP features, including:

[0115] Features of the clipping region obtained after clipping the bounding rectangle of the mask for:

[0116] ;

[0117] in, For CLIP image coding network, For the current keyframe image, `bbox` is the target instance mask, `crop` is the bounding box of the mask, `resize` is the image cropping function, and `resize` is the size scaling function to uniform size.

[0118] Mask purification features obtained by filling non-masked areas with a preset mean value for

[0119] ;

[0120] in, The mean of the ImageNet dataset;

[0121] Global image features corresponding to the complete image for:

[0122] ;

[0123] In the process of selecting the effective mask, if there is a prior 3D back projection mask, the segmentation result with the largest IoU with the 3D projection mask is selected; otherwise, if there is no prior, the mask with the highest prediction confidence of the model itself is selected.

[0124] Furthermore, the final fused single-target feature obtained by adaptively weighting and fusing the three CLIP visual features is described above. for:

[0125] ;

[0126] in, For the normalization of the second norm, For cropping feature weights, For global feature weights, Here, r represents the weight of the mask-refined features, and r is the proportion of the target image; where,

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] in, It is a double sigmoid basis function. The slope This represents the inflection point of the function.

[0132] A processing apparatus, comprising:

[0133] At least one memory for storing one or more programs;

[0134] At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor can perform the above-described method.

[0135] A readable storage medium for storing a computer program that, when executed by a processor, enables the implementation of the above-described method.

[0136] Compared with existing technologies, the object-level semantic mapping and workstation alignment method and storage medium provided by this invention can integrate geometric and semantic criteria for cross-frame association, reducing mismatches in similar experimental instruments; it adapts to different scale targets by adaptively fusing features of three CLIP (contrastive language image pre-trained model) based on mask area, avoiding feature effect decay; it uses dual anti-dilation constraints to limit the bounding box size of objects, preventing abnormal expansion of point cloud size; it corrects the deviation between the object map and the optimized pose through three-level pose synchronization optimization; it combines multi-view feature aggregation and 3D projection to complete occluded and missed objects; it uses prototype verification to screen abnormal objects, facilitating cleanup during export; and it achieves unified alignment of workstation coordinates based on image feature matching, so that the built map can directly support actual robot operations. Attached Figure Description

[0137] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0138] Figure 1 A schematic diagram illustrating the implementation flow of the method provided in this embodiment of the invention;

[0139] Figure 2 This is a schematic diagram of the structure of the three-channel CLIP feature area adaptive fusion module provided in an embodiment of the present invention;

[0140] Figure 3 A schematic diagram illustrating the implementation process of geometric-semantic joint cross-frame association and dual anti-expansion gating provided in an embodiment of the present invention;

[0141] Figure 4 A schematic diagram of the multi-view semantic feature library and three-dimensional projection feedback closed-loop structure provided in the embodiments of the present invention;

[0142] Figure 5 This is a schematic diagram illustrating the implementation process of workstation alignment based on rendering and feature matching provided in an embodiment of the present invention. Detailed Implementation

[0143] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific content of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, which do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0144] First, the following explanations are provided for the terms that may be used in this article:

[0145] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0146] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0147] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0148] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0149] When concentration, temperature, pressure, size, or other parameters are expressed as numerical ranges, such ranges should be understood to specifically disclose all ranges formed by any pairing of upper limits, lower limits, or preferred values ​​within that range, regardless of whether the range is explicitly stated; for example, if the numerical range "2 to 8" is stated, then that range should be interpreted to include ranges such as "2 to 7", "2 to 6", "5 to 7", "3 to 4 and 6 to 7", "3 to 5 and 7", "2 and 5 to 7", etc. Unless otherwise stated, the numerical ranges described herein include both their endpoints and all integers and fractions within that range.

[0150] The terms “center,” “longitudinal,” “lateral,” “length,” “width,” “thickness,” “upper,” “lower,” “front,” “back,” “left,” “right,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” “outer,” “clockwise,” and “counterclockwise” indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience and simplification of description and do not imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this document.

[0151] This invention provides an object-level semantic mapping and workstation alignment method, specifically an online object-level semantic mapping and workstation alignment method suitable for chemical laboratory scenarios. The aim is to ensure that the output map of this method possesses at least the following characteristics:

[0152] (1) Incrementally build online and attach semantic embeddings for each object;

[0153] (2) Cross-frame association utilizes both geometric and semantic criteria;

[0154] (3) Apply a hard upper bound constraint to the size of the object point cloud;

[0155] (4) Occlusion and missed detection are eliminated through multi-view weighted aggregation and 3D projection feedback;

[0156] (5) Identify spurious objects generated by positive feedback loops through category prototype post-processing;

[0157] (6) By using a workstation alignment mechanism based on feature matching between the rendered image and keyframes, the local high-precision scene is aligned to the object-level map and used for downstream robot operation tasks.

[0158] Based on the above objectives, embodiments of the present invention provide a corresponding online object-level semantic SLAM mapping and workstation alignment method for chemical laboratories. This method can jointly model online perception, association, aggregation, constraint and alignment, and output an object-level 3D map expressed in the robot navigation map coordinate system with semantic embedding, thereby solving the problems of existing methods in cross-frame association, feature aggregation, point cloud expansion control and coordinate system unification.

[0159] The online object-level semantic SLAM mapping (i.e., simultaneous localization mapping) and workstation alignment method for chemical laboratories provided in this embodiment of the invention requires object-level mapping and coordinate system alignment of the RGB-D video stream acquired by the mobile robot. The corresponding implementation process may include the following steps:

[0160] Step 1: RGB-D (color image data depth image) data acquisition and pose estimation;

[0161] Specifically, it can receive RGB-D frame sequences transmitted by the robot's depth camera, call the visual SLAM backend to perform pose estimation and tracking status judgment, and maintain the consistency of the built object point cloud with the latest optimized pose with a three-level pose correction mechanism, thereby obtaining the corresponding pose-corrected point cloud and object pose.

[0162] In step 1, the pose estimation can be implemented based on sparse feature visual SLAM, and the SE(3) pose of the camera relative to the world coordinate system in the current frame is output. When the tracking state is lost or short-term tracking, no subsequent semantic processing is performed on the frame, and the frame waits for internal relocalization within SLAM. When the tracking state recovers from lost to normal, the pose jump distance of this relocalization is recorded.

[0163] The consistency maintenance achieved by the three-level pose correction mechanism may include:

[0164] The first level monitors the "major change index" counter exposed by the SLAM system. This counter increments after loop closure fusion or global bundle adjustment. Once it changes, a correction transformation derived from the pose change of the corresponding contributing keyframe is applied to all object point clouds, and a global merging is triggered.

[0165] The second level involves iterating through the contribution keyframes of all objects every preset number of keyframes and applying the correction corresponding to the difference between their old and new poses to the point cloud of that object.

[0166] The third level involves performing a final full correction with a smaller drift threshold before the video stream ends (the mapping and observation process ends). This involves performing an overall pose transformation correction on all established 3D point clouds of objects, object centroid coordinates, and object aggregate semantic association poses within the global object map. The drift threshold is determined based on parameters such as scene scale, camera motion amplitude, cumulative pose error of the SLAM system, and normal position fluctuation range of the object.

[0167] Step 2: Object detection, instance segmentation, and adaptive fusion of three CLIP feature areas;

[0168] Based on the pose-corrected RGB-D image data and accurate spatial pose, open-domain object detection is performed on selected keyframes to obtain detection boxes containing the corresponding objects. Instance segmentation is then performed on the detection boxes to obtain pixel-level masks. Finally, mask clipping, mask masking, and full masking are extracted in parallel for each mask. Figure 3 The visual features of the path CLIP are obtained and weighted by smoothing the image using a double sigmoid (i.e., double sigmoid activation function) mapping according to the mask area ratio (i.e., the ratio of the mask pixel area to the total pixel area of ​​the whole frame image) to obtain the object-level semantic embedding of the observation, i.e., the object-level semantic feature vector.

[0169] Furthermore, in step 2, the open-domain target detection is trained with a predefined chemical instrument category as the target, and the category set is used as the sole source of category names in subsequent semantic comparisons; the instance segmentation model uses the detection box as a prompt and outputs multiple candidate masks. The mask selection rule is: when a corresponding 3D projection mask exists, the one with the largest 2D intersection-union ratio with the projection mask is selected; otherwise, the one with the largest predicted intersection-union ratio score is selected.

[0170] The three CLIP features are as follows:

[0171] The first path involves cropping the outer rectangle of the mask and then feeding it into the image encoder;

[0172] The second path fills the non-masked area with a preset mean value and then sends it to the image encoder;

[0173] The third path feeds the complete original image into the image encoder, and all masks in this frame share the features of this path.

[0174] The weighted fusion is based on the proportion of mask area. The calculation formula is: ,in, The total number of valid pixels within the segmentation mask for an object instance. The input image height in pixels. The input image width is given in pixels; the corresponding three weights are obtained by linear combination of two sigmoid basis functions with different centers and steepnesses, and the function construction method of this weight expression ensures that the three weights are always positive, sum to 1, and are consistent with... It is continuously differentiable everywhere.

[0175] Step 3: 2D to 3D backprojection and point cloud cleanup;

[0176] By utilizing depth maps and in-camera participation in camera pose, the effective pixels within each mask are back-projected from 2D to 3D to the world coordinate system. Voxel downsampling and statistical outlier removal are then performed on the back-projected point cloud to reduce the number of point clouds and remove noise and artifacts based on neighborhood distance, thus establishing the corresponding object-level point cloud.

[0177] In step 3, the back-projection operation is performed by the camera intrinsic parameter matrix. With camera pose The process is completed in batches on the GPU (Graphics Processing Unit); the point cloud cleaning includes two steps: voxel downsampling and statistical outlier removal, and the voxel resolution value is no greater than half of the minimum resolvable size of the instrument to be built.

[0178] Step 4: Geometric-semantic joint cross-frame association and dual anti-dilation gating;

[0179] Specifically, to integrate the established object-level point cloud update into the current global object map, a two-stage association is performed between the global object map and the current frame observation:

[0180] The first stage involves digesting the correspondences formed by the previous frame's 3D projection; prioritizing the association of existing stable objects by quickly reusing historical spatial projection matching results, thereby improving matching efficiency.

[0181] In the second stage, a joint score of geometric similarity and semantic similarity is constructed for the remaining observations and a greedy assignment is performed to handle new targets that were not matched in the first stage (i.e., the corresponding remaining observations), and precise pairing is performed by combining spatial morphology and semantic features to complete all the association relationships; wherein, the remaining observations refer to the predicted targets observed in the current frame but not appearing in the historical spatial projection.

[0182] During the two-stage operation described above, for each successfully matched object update action, it is also necessary to perform centroid drift gating and absolute volume gating based on robust baseline in sequence. If either gating fails, the object update action is rejected.

[0183] In the second stage described above, the geometric similarity is taken as the larger of the overlap of the three-dimensional point cloud and the intersection-union ratio of the two-dimensional projection mask; the semantic similarity is taken as the cosine similarity between the currently observed fusion feature and the candidate object aggregation feature; the joint score is a weighted sum of the two, and the correspondences exceeding the threshold are greedily selected in descending order of the scores.

[0184] During the above-mentioned dual anti-expansion gating process:

[0185] The centroid drift gating specifies that the Euclidean distance between the centroid of the new observation point cloud and the existing centroid of the candidate object cloud does not exceed a preset threshold.

[0186] The absolute volume gating limits the diagonal length of the axis-aligned bounding box of the merged point cloud to within a "robust baseline" multiplied by a preset multiple. The robust baseline is the median of the diagonal length of the bounding box corresponding to the object in the previous several observations.

[0187] If either of the two gating conditions fails, the object update operation is rejected. If the rejected observation has a confidence level higher than a preset threshold, it is created as a new object; otherwise, it is discarded.

[0188] Step 5: Maintenance of multi-view semantic feature library and 3D projection feedback loop closure;

[0189] The corresponding feature library maintenance includes: maintaining a feature library that retains several viewpoints for each object, and obtaining the aggregated semantic embedding of the object by using a weighted average based on cosine distance based on the feature library;

[0190] The corresponding 3D projection feedback loop includes: projecting the 3D point cloud of all confirmed objects back to the next keyframe image plane, forming a projection prompt box after visibility filtering, and sending it together with the detection box in step 2 into the instance segmentation model to complete object matching and map update, and finally confirming and incorporating it into the global map, thus forming a closed loop of "detection (detection box) - projection (projection prompt box) - confirmation".

[0191] In step 5, the multi-view feature library uses the pixel area of ​​the observation mask as a quality indicator and retains several high-quality views according to the Top-K strategy; the aggregation adopts a weighted average based on cosine distance, and in the weighted average processing, features that are consistent with the direction of most views are given higher weights.

[0192] The three-dimensional projection feedback is activated after the projection warm-up period, that is, after the pose accuracy meets the standard and the spatial mapping tends to be stable, the projection feedback mechanism is officially activated.

[0193] The process of forming a projection prompt box after visibility filtering may include: performing a depth consistency check between each projection mask and the current frame depth map, and forming a valid prompt box only when the proportion of visible points exceeds a preset threshold;

[0194] Furthermore, for pure projection cue boxes that are not covered by any detection boxes, a linear decay is applied according to their median projection depth to reduce their confidence, and the observation counts of "detection source" and "projection source" can be recorded separately in the object data structure.

[0195] Step 6: Periodic global merging and fragment timeout cleanup;

[0196] In this step, the object map needs to be traversed every preset number of keyframes to globally merge geometrically overlapping and semantically consistent object pairs, as well as object pairs of the same category and with adjacent centroids; and periodically clean up objects that have not been matched for a long time and have been observed very few times; for the global merging, the corresponding robust baseline is taken as the larger one so that the subsequent volume constraints are still effective after merging.

[0197] The global merge can include two paths:

[0198] Path 1: Dual threshold merging based on 3D point cloud overlap and cosine similarity of aggregated features;

[0199] Path 2 involves merging pairs of objects of the same category and with adjacent centroids.

[0200] Furthermore, for the robust baseline after global merging, the larger value of the two merged baselines can be taken so that subsequent volume constraints remain effective after merging.

[0201] The debris cleanup is only applied to unconfirmed objects and those observed very infrequently (e.g., no more than 1) to avoid mistakenly deleting legitimate objects.

[0202] Step 7: Final state merging and prototype verification after the session ends;

[0203] After the video stream ends, final-state merging is iteratively performed with a more lenient threshold to handle chained merging; the aggregate semantic embedding of each object is used to calculate the cosine similarity with the offline constructed category prototype (i.e., category standard semantic feature template), and combined with the 3D projection completion ratio (referred to as projection ratio), suspicious objects generated by the "detection-projection" positive feedback are identified and marked or removed; the ratio is the ratio of the effective pixels (effective area) restored by completion to the overall projection area;

[0204] Furthermore, the final state merging can use a lenient threshold (such as a larger allowed upper limit for centroid distance, lower geometric and semantic thresholds, etc.), and iteratively execute until no new merging is generated or the maximum number of iterations is reached. The lenient threshold refers to a threshold greater than a predetermined amount of the standard threshold. The category prototype is obtained by cropping the image corresponding to each category ground truth box (such as a manually labeled correct target bounding box) in the training set, encoding it with the same image encoder, calculating the mean, and then L2 normalizing it. It is constructed and stored offline. The suspicious object is determined as an object that meets at least one of the following two conditions: "prototype similarity is lower than a preset threshold" or "the proportion of observations from projection sources is higher than a preset threshold". When the prototype similarity is lower than the preset threshold, it indicates that the aggregated semantic features of the object observed in real time are too different from the offline standard category prototype (i.e., the corresponding aggregated semantic features of the object are likely to be misidentified). When the proportion of observations from projection sources is higher than the preset threshold, it indicates that most of the observation data of the object is not obtained from real-world detection, but from 3D projection calculation, i.e., it lacks real-world evidence and has low credibility.

[0205] Step 8: Workstation Alignment and Coordinate System I Based on Rendering and Feature Matching;

[0206] For each workstation, the original training image is loaded and matched with the selected SLAM keyframes in two stages of sparse and dense features under mask constraints to construct the correspondence between two-dimensional and two-dimensional and two-dimensional and three-dimensional. The seven-degree-of-freedom similarity transformation Sim(3) is solved by RANSAC (random sampling consensus algorithm) and Umeyama (Meishan algorithm), and optimized by nonlinear refinement and ICP (iterative nearest point algorithm) refinement with Gaussian point cloud physical size constraints. For multiple workstations corresponding to the same model of equipment, two rounds of greedy allocation are required for disambiguation and scene reuse. Finally, the calibration grid points in the coordinate system of each workstation are uniformly transformed to the robot navigation map coordinate system.

[0207] Furthermore, the feature matching can adopt a two-stage strategy, which mainly uses SuperPoint+LightGlue (superpoint feature + lightweight feature matching network) sparse matching and supplements it with RoMa (robust multi-view dense matching model) dense matching, and limits feature point extraction and matching to the bounding box region obtained by projecting the 3D point cloud onto the keyframe.

[0208] The Sim(3) estimation uses the RANSAC+Umeyama closed solution. When the physical spacing of the calibrated grid is known, the scale degree of freedom is locked as the ratio of the physical spacing to the mean distance of the grid in the local coordinate system.

[0209] The ICP refinement is activated as a degradation scheme when there are insufficient in-points in RANSAC. Its sampling radius on the SLAM side is given by multiplying the diagonal of the physical bounding box of the corresponding workstation's Gaussian point cloud by a preset coefficient, so as to avoid the interference of point cloud expansion on ICP.

[0210] The multi-instance disambiguation adopts a two-round greedy allocation: the first round strictly allocates according to the quality score at a 1:1 ratio, and the second round allows excess object instances to reuse the allocated workstation scene of the same model. The quality score is jointly determined by the number of in-points in RANSAC, the in-point rate, and the scale ratio deviation.

[0211] Compared with the prior art, the implementation scheme for online object-level semantic SLAM mapping and workstation alignment in chemical laboratories provided by the above embodiments of the present invention can at least include the following beneficial technical effects:

[0212] (1) By using both geometric and semantic criteria to participate in cross-frame association, the false association rate between chemical instruments of the same type can be reduced;

[0213] (2) The three CLIP features are adaptively fused according to the mask area, so that no additional training is required. For small targets, the mask is cropped, and for large targets, the whole image is fused, which effectively alleviates the degradation of single features at different scales.

[0214] (3) Introducing a dual anti-expansion gating based on robust baseline and absolute volume can apply a hard upper bound to the bounding box of the object point cloud, avoiding the failure of gating based on successive growth rate on objects with high observation count due to exponential accumulation.

[0215] (4) The three-level pose consistency maintenance mechanism can synchronously reflect the loop fusion, periodic local bundle adjustment and final state optimization results of the SLAM backend onto the built object point cloud, thereby reducing the deviation between the existing map and the latest optimized pose.

[0216] (5) A three-dimensional projection-detection feedback closed-loop processing scheme is adopted, namely, multi-view weighted aggregation combined with three-dimensional projection feedback, so that partially occluded or short-term missed objects can be further segmented and confirmed by the projection prompt box in subsequent frames;

[0217] (6) Introduce suspicious object identification under category prototype verification, that is, the prototype verification post-processing identifies suspicious objects generated by the positive feedback of "detection-projection", which is convenient for marking or removing them in the export stage.

[0218] (7) The workstation alignment mechanism based on feature matching between the rendering map and the keyframes will bring the coordinate system of each workstation and the coordinate system of the robot navigation map to the same reference system, so that the output map can directly provide the workstation-level target position for the downstream tasks of the robot.

[0219] To facilitate a further understanding of the embodiments of the present invention, the specific implementation methods of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0220] like Figure 1 As shown, the method provided in this embodiment of the invention may specifically include the following processing steps:

[0221] Step 11: RGB-D data stream acquisition and pose estimation

[0222] Let the first The RGB image of the frame is Depth map is The visual SLAM module performs pose estimation and tracking state determination on this frame, and outputs the camera's pose to the world.

[0223]

[0224] Introducing tracking state variables ;when When, no backprojection, association, or update operations are performed on the current frame; when and At that time, record the relocation event and the pose jump distance.

[0225] The backend of visual SLAM continuously optimizes keyframe poses during tracking, leading to a cumulative deviation between the already back-projected and stored object point cloud and the latest pose. To eliminate this deviation, for each object... Maintain its contribution keyframe set Apply the following formula to its point cloud:

[0226]

[0227]

[0228] in, This represents the increment of the K-pose change in the keyframe. To optimize the pose transformation matrix of the previous keyframe K, To optimize the latest pose matrix of keyframe K, The average rotation correction across all associated keyframes is used as the overall average rotation correction value for the object. for The split rotation increment, for Decompose the translation increment, The average translation correction across all associated keyframes is used as the overall translation correction value. To align the point coordinates with the latest pose after correcting the point cloud coordinates. The 3D point coordinates of the object before point cloud coordinate correction. For objects The set of contributing keyframes;

[0229] The above correction is triggered in three levels, namely:

[0230] Level 1: Monitor the "major change index" of the SLAM system. This index increments after loop fusion or global beam adjustment. Once it changes, a correction is performed on all objects, triggering a global merge (i.e., the global merge operation performed in subsequent step 16).

[0231] The second level, every... A periodic correction is performed once per keyframe, only for frames where the correction amount is greater than the periodic correction threshold. Update the objects (implementation can be done using...) , );

[0232] The third level uses a smaller drift threshold as the full correction threshold before the session ends (this can be implemented as follows). Perform a final full correction, i.e., the full correction threshold is less than the periodic correction threshold.

[0233] Step 12: Object detection, instance segmentation, and adaptive fusion of three CLIP feature areas

[0234] like Figure 2 As shown, for each selected keyframe Perform the following processing:

[0235] (a) Open domain target detection yields the detection set. ,in , , These are the bounding box, confidence score, and category name, respectively. This represents the total number of targets detected in the current frame.

[0236] (b) The detection boxes are fused with the 3D projection cue boxes given in subsequent step 15 according to the 2D intersection-union ratio (IoU) to obtain a unified set of cue boxes, which is then fed into the instance segmentation model. The segmentation model outputs multiple candidate masks, and the final mask is selected according to the following formula. for:

[0237]

[0238] If a 3D back-projection mask prior exists, the segmentation result with the largest IoU (Intersection over Union) with the 3D projection mask is selected; otherwise, if there is no prior, the mask with the highest prediction confidence of the model itself is selected.

[0239] in, A two-dimensional prior mask for backprojection of historical 3D point clouds. To determine the matching degree between the segmentation mask and the geometric prior, Confidence in the native mask for the segmentation model;

[0240] (c) For each effective mask (i.e., final mask) Parallel extraction of three CLIP features:

[0241] Method 1: The clipping region feature obtained after clipping the bounding rectangle of the mask (i.e., clipping the Masked Crop by bbos). for:

[0242]

[0243] in, For CLIP image coding network, For the current keyframe image, `bbox` is the target instance mask, `crop` is the bounding box of the mask, `resize` is the image cropping function, and `resize` is the size scaling function to uniform size.

[0244] Method 2: Mask-refined features obtained by filling the non-mask regions of the Masked Image with a preset mean. for:

[0245]

[0246] in, The mean of the ImageNet dataset;

[0247] Method 3: Global image features corresponding to the full image for:

[0248]

[0249] The above three features are input together into the OpenCLIP ViT-L / 14 image encoder (batch parallel) to obtain three raw features;

[0250] (d) Implement area adaptive weighting based on double sigmoid smooth mapping of r=|m| / (H·W); so that the weights of the three features change dynamically according to the target area ratio to adapt to objects of different sizes;

[0251] Furthermore, based on the proportion of mask area Define the double sigmoid basis function as the weight control variable. for:

[0252]

[0253] in, Here, r represents the image height and width, and r represents the target area's proportion of the image. Temperature is the temperature of the sigmoid function, used to control the rate of transition. These are inflection points of the function, used to divide size intervals;

[0254] When implementing, it can be taken Used to depict the transition from "small goals to medium goals" Used to depict the transition from "medium-sized target to large target". The three adaptive weights are:

[0255]

[0256]

[0257]

[0258] in, For cropping feature weights, For global feature weights, Weights for mask-based purification features;

[0259] It can be proven from the above construction and And mapping pairs Continuous and differentiable everywhere The feature weight coefficients are dynamically changed as the target area percentage r changes.

[0260] The corresponding fusion features are obtained by performing weighted fusion and L2 normalization. The corresponding single-target final fusion features for:

[0261]

[0262] in, Normalization to L2; through this fusion feature The three features can be fused with adaptive weights and normalized to obtain semantic features that are scale-uniform and adapted to the target size.

[0263] Final keyframe Two-dimensional observation set for:

[0264]

[0265] in, To test the confidence level, For the target category, Mark the source of the observation; through this two-dimensional observation set It can summarize the mask, features, confidence, category and source information of all targets in a single frame to output structured observation data.

[0266] Step 13: 2D to 3D backprojection and point cloud cleanup

[0267] Let the camera intrinsic parameter matrix be set. for

[0268]

[0269] in, The focal length of the camera in the x-direction. The focal length of the camera in the y-direction. , The pixel coordinates of the principal point of the image;

[0270] For each mask Set the effective pixel coordinates within By back-projecting the pixel onto the camera coordinate system using the following formula, the corresponding 3D point in the camera coordinate system can be obtained. for:

[0271]

[0272] Wherein, the pixel corresponds to the depth value Must fall within the effective depth range Internal (can be selected during implementation) and ), This is the depth map of the current frame;

[0273] Then from the pose Transform to three-dimensional space points in the world coordinate system ,for:

[0274]

[0275] in, For rotation matrix, It is a translation vector. This represents the coordinates of a point in three-dimensional space in the camera coordinate system; this formula can be used to uniformly transform the local coordinates of the camera to the global world coordinate system.

[0276] The point cloud obtained by the transformation Perform voxel downsampling to obtain a simplified point cloud after downsampling. for:

[0277]

[0278] And perform statistical outlier removal. This removal process is based on the number of statistical nearest neighbors of a single point and a distance threshold. Let k be the number of statistical nearest neighbors of a single point, such as... Distance threshold is This process removes outliers with excessive deviations, thereby improving the overall quality of the point cloud and the accuracy of modeling.

[0279] in, Let be the side length of the voxel cube. The average distance of the neighborhood. This represents the standard deviation of the distance.

[0280] Step 14: Geometric-semantic joint cross-frame association and dual anti-dilation gating

[0281] The corresponding geometric-semantic joint cross-frame association and dual anti-dilation gating process is as follows: Figure 3 As shown, this allows for the combination of spatial geometric location and semantic features of the object to complete the matching and association of the same object across different frames.

[0282] Furthermore, such as Figure 3 As shown, for the current frame observation set (instance segmentation mask + three-way fusion features + category), the corresponding association is performed in two stages;

[0283] Let the global object map be Each object maintain:

[0284]

[0285] in, A collection of three-dimensional point clouds of objects. For object category information, For multi-view feature library, As an aggregation feature, The cumulative number of observations of the object. To detect source observation counts, For the observation count of the projection source, As the robust size baseline for the object, Associate keyframes, poses, and other auxiliary information with objects, where j is the index of a single object and k is the total number of objects stored in the map.

[0286] Phase 1, Projection Pre-matching Phase:

[0287] In the detection box-projection box (i.e., Yolox box vs. projection box IoU) correspondence given in step 15, if the two-dimensional IoU between the two is greater than the threshold... (That is, the threshold for the intersection-union ratio of the 2D detection box and the projection box, which can be selected during implementation) If so, the observation-object correspondence is directly established, and then semantic verification is performed. (Optional during implementation) This will be confirmed later; among them, The cosine similarity between the currently observed features and the aggregated features of the object. The threshold for semantic similarity determination. Let i be the final fused semantic features of the single target in frame t. The aggregate feature of the j-th object in the map;

[0288] Through this stage of processing, we can quickly perform coarse matching based on historical projection priors, establish corresponding relationships after double condition verification, and reduce the amount of subsequent matching calculations.

[0289] Phase 2, Global Greedy Search Phase:

[0290] For the remaining unmatched observations With unoccupied candidate objects The similarity is defined as follows:

[0291] 3D point cloud overlap for:

[0292]

[0293] in, For the current observation target point cloud, For the current observation of three-dimensional points, For 3D points of objects on a map, , which is the threshold for spatial distance determination; the overlap of the three-dimensional point cloud can be used to count the proportion of close overlap between the current point cloud and the point cloud of map objects, representing the degree of overlap in three-dimensional space;

[0294] Two-dimensional projection mask IoU, i.e. for:

[0295]

[0296] in, For the target mask in the current frame, A two-dimensional mask obtained by backprojecting map objects;

[0297] Geometric similarity and semantic similarity are respectively:

[0298]

[0299]

[0300] The combined score is:

[0301]

[0302] in, These are the weighting coefficients;

[0303] according to Greedy selection in descending order exceeds the threshold for effective matching. (Can be selected during implementation) The correspondence between observations and objects is such that each observation is matched at most once.

[0304] Based on the processing results of Phase 1 and Phase 2, a dual anti-expansion gating strategy is implemented:

[0305] After each successful match Perform the following two checks:

[0306] Gating 1, center of mass drift gating:

[0307]

[0308] in, This represents the Euclidean offset distance between the centroids of the currently observed object and existing objects on the map. The current observation point cloud centroid, Set the centroid of the point cloud for the existing objects on the map. This is the allowable threshold for positional offset; based on this formula, the offset of the center of the old and new objects can be determined to avoid unreasonable drift of the object's position.

[0309] Gating 2, Absolute Volume Gating:

[0310]

[0311]

[0312] in, This represents the diagonal length of the 3D bounding box of the object after merging the old and new point clouds, used to characterize the overall spatial dimensions of the merged object. The overall point cloud after merging the old and new point clouds. This is the upper limit of the size magnification. The median length of the diagonal of the bounding box of the object in the first 5 observations (robust baseline).

[0313] If any gate fails, the object update is rejected; if rejected and the observation confidence is higher than a predetermined value, such as 0.6, it can be created as a new object; otherwise, it is discarded.

[0314] Step 15: Maintenance of Multi-view Semantic Feature Library and Closure of 3D Projection Feedback

[0315] like Figure 4 As shown, for each successful match Perform the following processing operations:

[0316] (a) Using the area of ​​the mask pixels As a quality indicator, Add to feature library Feature library Top-K perspective features containing this feature ( , ), For the k-th feature in the database, For the corresponding sample mask pixel area weights; and when (When implementing, it can be ordered) When removing The smallest element is the single-target feature sample with the smallest mask pixel area and the smallest effective imaging area.

[0317] (b) Perform cosine distance weighted aggregation as follows:

[0318]

[0319] in, The characteristic mean, The total number of samples in the object feature library. These are single-feature weighted coefficients. This is the exponential adjustment coefficient.

[0320]

[0321] The above processing can fuse similarity and area-weighted aggregated features, and after normalization, the final aggregated features of the object can be obtained. .

[0322] (c) The 3D projection feedback process includes: constructing a global object map O (point cloud + aggregated features), projecting the 3D point cloud onto the next keyframe image plane, visibility gating and depth consistency filtering, merging with YOLOX detection boxes into a SAM2 cue box set, and 3D projection feedback closed loop to achieve closed loop update;

[0323] Specifically, when the total number of keyframes exceeds the warm-up threshold (When implementing, it can be set to 30) After that, for all confirmed objects Each three-dimensional point in First, from the world in the current frame to the camera pose. Transform to camera coordinate system to obtain camera coordinates for:

[0324]

[0325] in, For rotation matrix, For world point coordinates, This is a translation vector; this formula can transform global 3D points to the current camera's local coordinate system.

[0326] Then, the pixel coordinates are obtained through perspective projection. Projected depth of that point :

[0327]

[0328] in, , The focal lengths of the camera in the x and y directions are... , , Let the coordinates of a 3D point be the three axes in the camera coordinate system. , The pixel offset coordinates of the principal point of the image;

[0329] The above processing enables the projection of a 3D point cloud onto the next keyframe image plane.

[0330] Generate a projection mask based on all projected pixels. With outer frame Let the number of projection points within the image be... Number of points with consistent depth (satisfying) The number of points, (where (v, u) is the actual observed depth value corresponding to pixel coordinates (v, u) in the t-th frame of the image) Then the visibility ratio for:

[0331]

[0332] Only when And the projection mask pixels and Only then can the projection mask form a valid projection cue box, and the projection confidence level is given according to the following depth attenuation. for:

[0333]

[0334] in, The median depth of the projection point. The maximum depth of the projection point;

[0335] The above processing achieves visibility gating and depth consistency filtering.

[0336] The projection box is fused with the target detection box of the next frame. That is, the filtered projection box and the YOLOX detection box are fused into a set of SAM2 cue boxes, which serve as the unified cue input for the instance segmentation model. The cue box segmentation obtains new observation features and point clouds, which are then fed back to update the feature library, realizing a three-dimensional projection feedback closed loop, thus forming a detection-projection-confirmation closed loop.

[0337] Step 16: Periodic global merging and fragment timeout cleanup

[0338] Keyframe intervals that trigger global merging Triggered once per keyframe (which can be set to 30 in implementation). (Global object merging operation) Iterate through all object pairs in the map. :

[0339] Path 1: Perform geometric-semantic dual threshold merging:

[0340]

[0341] in, The degree of overlap of the three-dimensional point clouds of the two objects. This is the threshold for merging 3D overlap. The threshold for semantic similarity merging. Cosine similarity of aggregate features between two objects To merge two objects into one, the following can be set during implementation: , ;

[0342] Path 2: Merge centroids of similar types:

[0343]

[0344] in, , For the two object category labels, , Let the point cloud centroids of the two objects be... This is the centroid distance merging threshold; during implementation, it can be set as follows: ;

[0345] Robust baseline during merging Take a conservative value:

[0346]

[0347] in, This formula establishes robust dimensional baselines for both objects; it ensures subsequent... (Updated) Still subject to effective volume constraints.

[0348] Each cleanup triggers a frame interval Each keyframe (which can be set to 15 during implementation) triggers a fragment cleanup;

[0349] object A item is considered cleanable if it meets all three of the following conditions:

[0350]

[0351] When implementing this, it can be ordered that: ;in Represents objects The confirmation threshold has been reached (when implementing this, the number of observations should be no less than 3). The cumulative number of observations of the object. This represents the current total number of keyframes. The keyframe count for the last time the object was matched.

[0352] Step 17: Iterative final state merging and prototype verification phantom suppression after session end

[0353] After the video stream ends, iterative final state merging is performed first. With a more lenient threshold (in practice, the final centroid distance can be combined with a more lenient threshold), the threshold can be adjusted. Final state 3D overlap merging with a relaxed threshold Final state semantic similarity merging with a relaxed threshold Repeatedly perform the merge until no new merges are generated or the maximum number of iterations is reached (which can be set to 10 rounds in practice). The corresponding more lenient threshold is defined as a threshold that is greater than the corresponding standard threshold by a predetermined amount.

[0354] Prototype validation is then performed, and the corresponding prototype validation process may include:

[0355] Assume offline training set for each object category Constructed category prototype for:

[0356]

[0357] in, For the set of all training samples corresponding to category c, This refers to the image and its corresponding detection box sample. For image feature extraction functions;

[0358] For each object Calculate prototype similarity With projection ratio for:

[0359]

[0360] in, The aggregated feature obtained after weighted fusion of the j-th object. For objects Category The number of observations obtained from projection matching. The number of observations directly detected by the detector;

[0361] object Under confirmed conditions, an object is deemed suspicious if either "prototype similarity is below a threshold" or "the proportion of observations from the projection source is above a threshold." The method for determining it is as follows:

[0362]

[0363] in, For objects The status has been confirmed. The threshold for determining prototype similarity. This is the threshold for determining the projection ratio; objects deemed suspicious are marked or removed during export. For logical AND, It is a logical OR; moreover, in implementation, the prototype similarity threshold can be set as follows: Threshold for determining projection ratio ;

[0364] Objects deemed suspicious can be marked or removed during export to cut off false objects generated by the "detection-projection" positive feedback loop.

[0365] Step 18: Workstation Alignment and Unified Coordinate System Output Based on Rendering and Feature Matching

[0366] like Figure 5 As shown, let the local Gaussian reconstruction scene corresponding to a certain workstation be... Its original training image set is ,in, For the nth original image, Let n be the camera pose transformation matrix for the nth frame. Let n be the camera intrinsic parameter matrix. This represents the total number of original training images in this scenario.

[0367] Reference Figure 5 As shown, the following processing flow can be executed for each established workstation:

[0368] (a) ORB keyframe selection, i.e., sorting by distance and angle to select keyframes;

[0369] Specifically, this can be done by analyzing the SLAM keyframe set according to distance and orientation angle. The frames are sorted according to a reference viewpoint, and then the corresponding optimal matching keyframes are selected. ;

[0370] (b) Load the CL-GS (hierarchical Gaussian sputtering reconstruction model) rendering map and implement the 3D point cloud projection mask (projection mask region) constraint;

[0371] Specifically, based on the loaded CL-GS rendering map, the 3D point cloud of the corresponding workstation object can be projected onto the optimal matching keyframe. Obtain the bounding box At the same time The rendered image corresponding to the Gaussian center projection onto the Gaussian reconstruction scene. Get the bounding box Subsequent feature point extraction and matching only occur during... and Execution within;

[0372] (c) Sparse feature matching SuperPoint+LightGlue (sparse feature detection descriptor algorithm + lightweight feature matching algorithm), and dense matching enhancement RoMa (dense feature matching algorithm), which is enabled when inlier (matching inlier, effective matching point) is not found;

[0373] Phase 1 uses SuperPoint+LightGlue for sparse feature matching; if the number of points in RANSAC (random sampling consistency, removing out-of-match points) is lower than a threshold, Phase 2 is initiated, using RoMa dense matching for enhancement; finally, a two-dimensional to two-dimensional correspondence set is obtained. ,in, Let L be the coordinates of the l-th two-dimensional pixel on the Gaussian rendering map. To match the coordinates of the l-th two-dimensional pixel on the keyframe;

[0374] (d) Construct a two-sided depth back projection from two-dimensional to three-dimensional correspondence;

[0375] Specifically, based on the two-dimensional to two-dimensional correspondence set Constructing a 3D-to-3D corresponding set from the back projection of the depth maps on both sides. ,in, For the l-th group of Gaussian scene, match the corresponding 3D spatial points. For SLAM mapping, the l-th group of corresponding 3D spatial points is matched; the corresponding two sides include: the SLAM side and the Gaussian side; wherein, the SLAM side is generated by D455 depth backprojection and pose mapping. (The pose transformation matrix from the camera coordinate system to the global world coordinate system) is transformed to the global coordinate system, and the approximate depth map constructed by the Gaussian side by the Gaussian center projection + splat radius (i.e. Gaussian sputtering rendering sampling radius) is back-projected to the workstation local coordinate system.

[0376] (e) RANSAC+Umeyama (Random Sample Consensus Algorithm + Umeyama Algorithm) for estimating the three-dimensional similarity transformation group The scale (spatial scaling factor) can be locked to the physical spacing of the calibration grid;

[0377] Specifically, for seven-degree-of-freedom similarity transformations Estimate the objective function for:

[0378]

[0379] in, These are the transformation scale factor, the 3D rotation matrix, and the 3D translation vector, respectively. For the Gaussian side three-dimensional matching point, For SLAM side 3D matching points;

[0380] The closed-form solution using RANSAC+Umeyama is given. Initial value; when the physical spacing of the calibration grid is known. At that time, the scale degree of freedom (i.e., the spatial scaling factor) will be used. Locked as:

[0381]

[0382] in, This represents the average distance between grid points in the local coordinate system.

[0383] Execute subsequent processing steps (f)-(i) to achieve GS physical size constraint ICP refinement + PnP cross-validation; determine the final result after optimization, correction and verification processes. Values, and thus determine the corresponding values. .

[0384] (f) For RANSAC solutions ( Perform Levenberg-Marquardt nonlinear refinement, parameterizing as follows: To finely adjust the corresponding value;

[0385] (g) When there are enough RANSAC interior points, perform ICP (Iterative Nearest Point) refinement; otherwise, perform ICP degradation. The sampling radius of the dense point cloud for ICP on the SLAM side. The physical bounding box diagonal of the Gaussian point cloud is given as follows:

[0386]

[0387] in, , Let be the diagonal length of the physical bounding box of the Gaussian point cloud. Let the physical scale coordinates of the Gaussian point cloud be... To calibrate the scale of the grid, These are the coordinates in the local coordinate system of the Gaussian model.

[0388] Through this process (g), during the ICP registration iteration, the distance error between corresponding points of two point clouds is continuously minimized, and the rotation matrix R and translation vector t are updated synchronously and iteratively to complete the parameter correction; the sampling radius only defines the matching search range and constrains the registration area, indirectly ensuring the effectiveness of R and t correction;

[0389] (h) with right (i.e., the aforementioned) Independent cross-validation was performed, where, This is a RANSAC-based PnP (2D-3D pose estimation) solution in OpenCV. PnP is a perspective n-point problem for finding the camera pose.

[0390] This process (h) allows for independent verification of whether R and t are correct and reliable.

[0391] (i) Disambiguate multiple instances of the same category corresponding to multiple workstations and construct a quality score. for:

[0392]

[0393] in, The number of interior points in RANSAC. For interior point proportions, For scale ratio;

[0394] It performs two rounds of greedy allocation: the first round strictly allocates object instances to workstation scenes at a 1:1 ratio; the second round, when the number of object instances exceeds the number of workstations, allows the remaining objects to reuse already allocated workstation scenes of the same model, with the allocation key set to... To maintain uniqueness, among which, This is the name of the workstation scene. Number the object instances; through this process (i), the final optimal R and t can be selected from multiple candidate results.

[0395] (J) Implement output from the workstation coordinate system to the global map coordinate system; specifically, this may include:

[0396] Based on the correspondence between the laser poses of robot navigation waypoints and the corresponding SLAM camera poses, a least-squares fitting two-dimensional similarity transformation is performed:

[0397]

[0398] in, The coordinates are in the robot's navigation map coordinate system. The scale factor for SLAM to map. It is a two-dimensional rotation matrix. For rotation angle, The horizontal coordinates in the SLAM coordinate system It is a two-dimensional translation vector;

[0399] Moreover, the SLAM coordinate system is used here. Quantity ( (for the height axis)

[0400] The transformation and the result obtained from step 18, process (e)-(i) Composite means that the calibration grid points in the coordinate system of each workstation (49 points can be selected for each workstation in implementation) can be uniformly transformed to the robot navigation map coordinate system.

[0401] The specific implementation methods provided in this invention can break down the object-level mapping process of a continuous RGB-D stream into eight steps: acquisition, detection and segmentation, back projection, association, aggregation, merging, final state post-processing, and workstation alignment. Each step provides clear input, output, and constraint conditions. The steps are connected through intermediate results such as object maps, feature libraries, and 3D projection prompts. The final output is an object-level 3D map expressed in the robot navigation map coordinate system and with semantic embedding.

[0402] Furthermore, the application process of the present invention will be described below with a specific application example.

[0403] In this application embodiment, the implementation scheme provided by the present invention is applied to a chemical laboratory scenario. Several workstations (such as purification workstations, liquid injection workstations, magnetic stirrer arrays, drying ovens, etc.) are deployed within the laboratory scenario. Each workstation corresponds to a local 3D scene pre-constructed using an offline Gaussian splashing algorithm. The robot body is equipped with a RealSense D455 depth camera with a resolution of 1280×720 and a depth measurement range of [missing information]. Depth quantization is performed as a 16-bit unsigned integer (unit: mm), with intrinsic parameters. , , , The robot transmits the RGB-D stream to the server via an SSH tunnel at approximately 5fps.

[0404] Step 1: Data acquisition and pose estimation;

[0405] The cloud receiving thread receives each frame in the format of "24-byte binary header + JPEG compressed RGB + PNG compressed 16-bit depth" and writes it into a two-level buffer of "memory queue + disk overflow" to avoid the receiving thread being blocked by the computing thread.

[0406] The visual SLAM module uses ORB-SLAM3's Python bind as its backend, loading the D455 camera configuration (pinhole model, Camera.bf=61.06) and ORB word-bag. When the backlog exceeds 100 frames, the keyframe interval is automatically increased by a factor of 4 (i.e., semantics are processed once every 8 frames) to reduce semantic density in exchange for real-time performance.

[0407] In an actual session with 3263 frames / 662 keyframes, the second-level Local BA cycle refresh was triggered a total of 12 times, with a maximum drift correction of about 30.8mm, exceeding the voxel resolution by 20mm; the third-level final state correction was triggered once, with a residual drift of about 6.8mm.

[0408] Step 2 involves detection, segmentation, and fusion of three CLIP features.

[0409] The open-domain detector is based on YOLOX-L (depth=1.0, width=1.0) and trained according to the 92 categories of chemical instruments defined in the ChemDataset92 dataset (10800 training samples + 2600 validation samples). The validation set AP@[0.50:0.95] is 0.994, and the single-frame inference time is about 4.42ms.

[0410] The instance segmentation model uses SAM2, with multimask_output enabled to output 3 candidate masks;

[0411] The three CLIP features are based on OpenCLIP ViT-L / 14 (OpenAI pre-trained weights, feature dimensions) This is an image encoder that takes the entire image of the current frame, each mask crop, and each mask stitched together into a single batch and feeds it into the encoder, accelerating it with blending precision using torch.amp.autocast; the area adaptive weights have the following extreme values ​​on both sides:

[0412] hour ;

[0413] hour ;

[0414] In this embodiment, the detection and segmentation results were experimentally determined to completely outline the workstation's contour based on the detection box, thus realizing a complete perception process from "coarse positioning (detection box)" to "fine segmentation (pixel-level mask)," verifying the effectiveness of the algorithm in laboratory industrial equipment scenarios.

[0415] Step 3: Back projection and point cloud cleanup;

[0416] voxel resolution Outlier removal is achieved by using statistical methods. Nearest neighbor, threshold .

[0417] Step 4: Association and Gating)

[0418] Phase 2 Global Greedy Search Threshold The weights of both geometric and semantic terms are taken as follows: ;

[0419] The 3D overlap calculation is accelerated using torch, and random sampling is performed on the large point cloud (the upper limit of the observation point cloud is 5000 points, and the upper limit of the object point cloud is 10000 points).

[0420] The centroid drift threshold is fixed at The robust baseline for volume constraints is given by the median of the diagonal length of the bounding box from the first 5 observations, multiplied by [a certain factor]. ;

[0421] For a balance observed 226 times, if the old gating method based on a successive growth rate of 1.3 is used, its theoretical bounding box upper limit can reach... In practice, it is uncontrollable; after changing to absolute volume gating, the diagonal of the bounding box is stably limited to within 3 times the robust baseline.

[0422] Step 5: Multi-view feature library and projection feedback;

[0423] Feature library capacity polymerization temperature ;

[0424] In the experiment, for the multi-view feature library, each object retains no more than 10 views corresponding to 768-dimensional CLIP features; projection warm-up period Keyframes, depth consistency threshold Visibility ratio lower bound ;

[0425] Specifically, after initial stabilization is achieved during the projection warm-up period (the first 30 keyframes), the system performs object detection and segmentation on consecutive keyframes of the workstation scene, and stores 768-dimensional CLIP features from up to 10 representative viewpoints for each object in the feature library; in map construction, a depth consistency threshold of 0.10m and a visibility ratio lower bound of 0.15 are used to filter noisy observations, thereby achieving robust object association and map construction.

[0426] Step 6: Global merging and fragment cleanup;

[0427] Merging intervals of 30 keyframes, geometric thresholding semantic threshold Upper limit of distance between centroids of the same type The fragmentation cleanup interval is 15 keyframes, with a timeout threshold. Only unconfirmed objects that have been observed no more than once are cleared.

[0428] Implement step 7: final state merging and prototype verification;

[0429] Final state merging , , Iterative execution, up to 10 rounds;

[0430] The category prototypes are obtained by cropping up to 50 ground truth boxes for each category from ChemDataset92, encoding them with ViT-L / 14, averaging the results, and then normalizing them using L2. These prototypes are then saved offline. Suspicious object threshold. , .

[0431] Step 8: Workstation alignment and coordinate system output;

[0432] A two-stage strategy of SuperPoint+LightGlue sparse matching+RoMa dense matching is adopted for feature matching. The mask constraint is given by the two-dimensional bounding box obtained by projecting the 3D point cloud of the constructed workstation object onto the key frame.

[0433] Sim(3) estimates using the RANSAC+Umeyama closed-form solution and locks the scale to the corresponding value of the physical spacing of the calibration grid; then Levenberg-Marquardt is used in... Nonlinear refinement is performed on the upper part; the sampling radius of ICP on the SLAM side is taken as the diagonal of the physical bounding box of the corresponding Gaussian point cloud of the workstation. times;

[0434] Multi-instance disambiguation employs a two-round greedy allocation; when a certain YOLOX category corresponds to multiple workstations, the first round performs a strict 1:1 allocation based on quality scores, and the second round allows excess objects to reuse the same model of workstation scenario.

[0435] Experimental results show that in this embodiment, the 49 calibration grid points of each workstation are transformed to the robot navigation map coordinate system in the feature matching results and calibration grid point reprojection verification. For different viewpoint images of the industrial workstation, the feature pairs obtained by SuperPoint+LightGlue sparse matching, in which the effective feature points matched in the first stage are provided with a robust initial correspondence by subsequent RoMa dense matching and Sim (3) pose estimation, laying the foundation for workstation alignment. The calibration grid points on the industrial workstation have been reprojected to the current keyframe. The average grid point spacing is 6.2px, the coefficient of variation CV=0.053, and the distance from the camera to the workstation d=2.07m, which verifies the accuracy of the aligned pose and the 3D map, proving that the workstation has been successfully calibrated to the robot navigation map coordinate system.

[0436] After the entire mapping and alignment process is completed, the exported data in step 8 includes:

[0437] (i) Semantic coloring point cloud;

[0438] (ii) Original RGB shaded point cloud;

[0439] (iii) 768-dimensional CLIP embedding for each object;

[0440] (iv) Object metadata, including fields such as point count, current bounding box diagonal, robust baseline, total number of observations, number of observations from detection sources, number of observations from projection sources, projection ratio, prototype similarity, and suspicious markers;

[0441] (v) Camera to world pose for all frames;

[0442] (vi) Transformation matrix from SLAM coordinate system to navigation map coordinate system;

[0443] (vii) Workstation alignment results.

[0444] The output object-level map and the calibration points of each workstation are located in the same robot navigation map coordinate system. It can be used as a basis for obtaining the target position of the workstation when the robot performs experimental tasks, and can also be used as the basic data for subsequent open vocabulary query and cross-task reuse.

[0445] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0446] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for object-level semantic mapping and workstation alignment, characterized in that, include: Receive color image data (RGB-D data) from the robot's depth camera, and call the Visual Simultaneous Localization and Mapping (SLAM) backend to perform pose estimation, obtaining the corresponding RGB-D data and spatial pose; Based on the obtained RGB-D data and spatial pose, open domain target detection is performed on the selected keyframes to obtain the corresponding detection boxes, and the detection boxes are segmented into instances to obtain pixel-level masks. For each of the pixel-level masks, mask cropping and mask masking are extracted in parallel, and these are combined with the full image of the key frame to form the CLIP visual features of the three-way image pre-training model. The CLIP visual features of the three-way image pre-training model are adaptively weighted and fused according to the proportion of mask pixel area, so as to obtain the object-level semantic embedding of the observation corresponding to the mask. By utilizing depth maps and in-camera participation in camera pose, the effective pixels within each mask are back-projected from two dimensions to three dimensions to the world coordinate system to establish an object-level point cloud. The object-level point cloud is assigned to the correspondence formed by the previous frame's 3D projection. A joint score of geometric similarity and semantic similarity is constructed for the remaining observations and a greedy allocation is performed. After the object update in the global object map is completed, the object-level map is obtained. Based on the object-level mapping, the local Gaussian reconstruction scene of each workstation and the selected keyframe are matched with sparse and dense features under mask constraints to construct the correspondence between two-dimensional and two-dimensional and two-dimensional and three-dimensional. The calibration grid points in each workstation coordinate system are then uniformly transformed to the robot navigation map coordinate system.

2. The object-level semantic mapping and workstation alignment method according to claim 1, characterized in that, The method further includes: point cloud cleaning of the object-level point cloud, and maintenance of a multi-view semantic feature library and closed-loop processing of 3D projection feedback; wherein, The point cloud cleaning process includes: Voxel downsampling is performed on the back-projected point cloud, and noise and artifacts are removed based on neighborhood distance. The maintenance process of the multi-perspective semantic feature library includes: For each object, a feature library is maintained that preserves several viewpoints, and the aggregated semantic embedding of the object is obtained by using a weighted average based on cosine distance based on this feature library. The three-dimensional projection feedback closed-loop processing process includes: Once the total number of keyframes exceeds the warm-up threshold, for each 3D point in all confirmed objects, the camera coordinates are first obtained by transforming the world position of the current frame to the camera pose and then to the camera coordinate system. for: ； in, For rotation matrix, For world point coordinates, It is a translation vector; Then, the pixel coordinates are obtained through perspective projection. Projected depth of that point : ； in, , The focal lengths of the camera in the x and y directions are... , , Let the coordinates of a 3D point be the three axes in the camera coordinate system. , The pixel offset coordinates of the principal point of the image; Generate a projection mask and bounding box based on all projected pixels, and count the number of projected points within the image. The number of points with consistent depth is Then the visibility ratio for: ； Only when And the projection mask pixels and At that time, the projection mask forms a valid projection cue box, and the corresponding projection confidence level... for: ； in, The median depth of the projection point. The maximum depth of the projection point; The projection prompt box is fused with the target detection box of the next frame and serves as a unified prompt input for the instance segmentation model, thus forming a closed loop of detection-projection-confirmation.

3. The object-level semantic mapping and workstation alignment method according to claim 2, characterized in that, This method also includes a periodic global merging and fragment timeout cleanup process, wherein: The process of periodic global merging includes: Every preset number of keyframes, the object map is traversed. Based on the dual threshold judgment rule of 3D point cloud overlap and aggregated feature cosine similarity, global merging is performed on object pairs that are geometrically overlapping, semantically consistent, of the same category and with adjacent centroids. Furthermore, for the robust baseline after global merging, the larger value of the two merged baselines is selected. The fragment timeout cleanup process includes: Periodically clean up objects that have not been matched for a long time and have been observed less than the predetermined number.

4. The object-level semantic mapping and workstation alignment method according to claim 2, characterized in that, The method also includes a final state merging and prototype verification process after the session ends, wherein: The final state merging process after the session ends includes: The final state merging is performed iteratively using a relaxed threshold until no new merges are generated or the maximum number of iterations is reached. The relaxed threshold is a threshold that is greater than a predetermined amount of the standard threshold. The relaxed threshold includes: a relaxed threshold for merging the centroid distance of the final state, a relaxed threshold for merging the three-dimensional overlap of the final state, and a relaxed threshold for merging the semantic similarity of the final state. The prototype verification process includes: Assume offline training set for each object category Constructed category prototype for: ； in, For L2 norm normalization, For the set of all training samples corresponding to category c, This refers to the image and its corresponding detection box sample. For image feature extraction functions, For image cropping operations; For each object Calculate prototype similarity With projection ratio for: ； in, For cosine similarity, The aggregated feature obtained after weighted fusion of the j-th object. For objects Category For objects Category The category prototype feature vector, The number of observations obtained from projection matching. The number of observations directly detected by the detector; object Given the confirmed information, determine whether the object is abnormal or suspicious. The method for determining it is as follows: ； in, For objects The status has been confirmed. The threshold for determining prototype similarity. This is the threshold for determining the projection ratio; objects deemed suspicious are marked or removed during export. For logical AND, For logical OR.

5. The object-level semantic mapping and workstation alignment method according to any one of claims 1 to 4, characterized in that, The pose estimation is implemented based on a three-level pose correction mechanism, and includes: The first level monitors the large change index counter exposed by the SLAM system. This counter increments after loop closure fusion or global bundle adjustment. When it changes, a correction transformation derived from the pose change of the corresponding contributing keyframe is applied to all object point clouds, and a global merging is triggered. The second level involves iterating through the contribution keyframes of all objects every preset number of keyframes, updating and correcting objects whose correction amount is greater than the periodic correction threshold, so that the correction corresponding to the difference between their old and new poses is applied to the point cloud of that object. The third level involves updating and correcting objects whose periodic correction amount is greater than the full correction threshold before the end of the mapping and observation process, whereby the periodic correction threshold is greater than the full correction threshold. And the update correction includes: for each object Maintain its contribution keyframe set And apply the following formula to its point cloud: ； ； in, This represents the increment of the K-pose change in the keyframe. To optimize the pose transformation matrix of the previous keyframe K, To optimize the latest pose matrix of keyframe K, The average rotation correction across all associated keyframes is used as the overall average rotation correction value for the object. This is an orthogonalization process based on singular value decomposition. for The split rotation increment, for Decompose the translation increment, The average translation correction across all associated keyframes is used as the overall translation correction value. To align the point coordinates with the latest pose after correcting the point cloud coordinates. The 3D point coordinates of the object before point cloud coordinate correction. For objects The set of keyframes that contribute to the project.

6. The object-level semantic mapping and workstation alignment method according to claim 5, characterized in that, The process of updating objects in the global object map includes: The projection pre-matching stage, which assigns the object-level point cloud to the corresponding relationship formed by the previous frame's 3D projection, includes: In the correspondence between the detection box and the projection box, if the Intersection over Union (IoU) of the two is greater than the threshold value of the IoU between the two-dimensional detection box and the projection box... Then, the observation-object correspondence is directly established, and then semantic verification is performed. This was subsequently confirmed; among them, The cosine similarity between the currently observed features and the aggregated features of the object. The threshold for semantic similarity determination. Let i be the final fused semantic features of the single target in frame t. The aggregate feature of the j-th object in the map; The global greedy search phase, which constructs a joint score of geometric and semantic similarity for the remaining observations and performs a greedy assignment, includes: For the remaining unmatched observations With unoccupied candidate objects The similarity is defined as follows: 3D point cloud overlap for: ； in, For the current observation target point cloud, For the current observation of three-dimensional points, For 3D points of objects on a map, A collection of three-dimensional point clouds of objects. The threshold for spatial distance determination. To obtain the minimum value; The IoU of a 2D projection mask is The calculation method is as follows: ； in, For the target mask in the current frame, A two-dimensional mask obtained by backprojecting map objects; Geometric similarity With semantic similarity They are respectively: ； ； Joint rating for: ； in, These are the weighting coefficients; according to Greedy selection in descending order exceeds the threshold for effective matching. The correspondence is such that each observation is matched with each object at most once; It also includes performing a dual anti-inflation gating process, namely: After each successful match, the current observed point cloud is used. Update existing objects on the map Data If so, the following two checks will be performed: Inspection based on centroid drift gating: ； in, This represents the Euclidean offset distance between the centroids of the currently observed object and existing objects on the map. The current observation point cloud centroid, Set the centroid of the point cloud for the existing objects on the map. This is the allowable threshold for positional offset; based on this formula, the offset of the center of the old and new objects can be determined to avoid unreasonable drift of the object's position. Inspection based on absolute volume gating: ； ； in, This represents the diagonal length of the 3D bounding box of the object after merging the old and new point clouds, used to characterize the overall spatial dimensions of the merged object. The overall point cloud after merging the old and new point clouds. This is the upper limit of the size magnification. The median length of the diagonal of the bounding box observed for this object; If any gate fails, the object update is rejected; if rejected and the observation confidence is higher than the predetermined value, it is created as a new object; otherwise, it is discarded.

7. The object-level semantic mapping and workstation alignment method according to claim 5, characterized in that, After constructing the correspondences from two dimensions to two dimensions and from two dimensions to three dimensions, the process also includes determining the seven-degree-of-freedom similarity transformation, and this process includes: Seven-degree-of-freedom similarity transformation Estimate the objective function for: ； in, For the index of the matching point pair, These are respectively the scale degrees of freedom, the three-dimensional rotation matrix, and the three-dimensional translation vector. For the Gaussian side three-dimensional matching point, For SLAM side 3D matching points; The closed-form solutions of the Random Sampling Consensus Algorithm (RANSAC) and the Umeyama algorithm are given. Initial value; when the physical spacing of the calibration grid is known. At that time, the scale degrees of freedom Locked as: ,in, This represents the average distance between grid points in the local coordinate system. Regarding the The initial value is refined using the Levenberg-Marquardt nonlinear algorithm, and parameterized as follows: ,in, For logarithmic mapping, This is the transformation from a rotation matrix to a rotation vector. It is a 7-dimensional real number space; The ICP registration iterative process is executed to minimize the distance error between corresponding points in the two point clouds, and the rotation matrix R and translation vector t are updated iteratively and synchronously; and the sampling radius of the dense point cloud of ICP on the SLAM side is... The physical bounding box diagonal of the Gaussian point cloud is given as follows: ； in, , Let be the diagonal length of the physical bounding box of the Gaussian point cloud. Let the physical scale coordinates of the Gaussian point cloud be... To calibrate the scale of the grid, These are the coordinates in the local coordinate system of the Gaussian model. by Seven-degree-of-freedom similarity transformation Perform independent cross-validation, where... This is a solution for PnP based on RANSAC in OpenCV, where PnP is a perspective problem with n points. Disambiguation of multiple instances corresponding to the same category of workstations, and construction of quality scores. for: ； in, The number of interior points in RANSAC. For interior point proportions, For scale ratio; It performs two rounds of greedy allocation: the first round strictly allocates object instances to workstation scenes at a 1:1 ratio; the second round, when the number of object instances exceeds the number of workstations, allows the remaining objects to reuse already allocated workstation scenes of the same model, with the allocation key set to... To maintain uniqueness, among which, This is the name of the workstation scene. Number the object instances; select the final optimal R and t from multiple candidate results to determine the final seven-degree-of-freedom similarity transformation. ; The process of uniformly transforming the calibration grid points in each workstation coordinate system to the robot navigation map coordinate system includes: Based on the correspondence between the laser poses of robot navigation waypoints and the corresponding SLAM camera poses, a least-squares fitting two-dimensional similarity transformation is performed: ； in, The coordinates are in the robot's navigation map coordinate system. The scale factor for SLAM to map. It is a two-dimensional rotation matrix. For rotation angle, The horizontal coordinates in the SLAM coordinate system It is a two-dimensional translation vector; The least squares fitting two-dimensional similarity transformation is compared with the final seven-degree-of-freedom similarity transformation. The coordinates of each workstation are combined and the calibration grid points in each workstation coordinate system are uniformly transformed to the robot navigation map coordinate system.

8. The object-level semantic mapping and workstation alignment method according to claim 5, characterized in that, The process of forming the visual features of the three-way image pre-training model CLIP involves processing each effective mask... Parallel extraction of three CLIP features, including: Features of the clipping region obtained after clipping the bounding rectangle of the mask for: ； in, For CLIP image coding network, For the current keyframe image, `bbox` is the target instance mask, `crop` is the bounding box of the mask, `resize` is the image cropping function, and `resize` is the size scaling function to uniform size. Mask purification features obtained by filling non-masked areas with a preset mean value for ； in, The mean of the ImageNet dataset; Global image features corresponding to the complete image for: ； In the process of selecting the effective mask, if there is a prior 3D back projection mask, the segmentation result with the largest IoU with the 3D projection mask is selected; otherwise, if there is no prior, the mask with the highest prediction confidence of the model itself is selected. Furthermore, the final fused single-target feature obtained by adaptively weighting and fusing the three CLIP visual features is described above. for: ； in, For the normalization of the second norm, For cropping feature weights, For global feature weights, Here, r represents the weight of the mask-refined features, and r is the proportion of the target image; where, ； ； ； ； in, It is a double sigmoid basis function. The slope This represents the inflection point of the function.

9. A processing device, characterized in that, include: At least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, such that when the one or more programs are executed by the processor, the processor can perform the method according to any one of claims 1-8.

10. A readable storage medium for storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement the method described in any one of claims 1-8.

Images