Image processing method and system for ORB-SLAM3

Abstract

The application discloses an image processing method and system for ORB-SLAM3 and belongs to the technical field of visual image processing and robot navigation. The method comprises the following steps: obtaining a detection result according to an RGB image in an input current image frame; obtaining a morphological mask according to the detection result; obtaining a posture key point according to the morphological mask; constructing a skeletal segment of each part according to the posture key point of each preprocessed ROI image; adopting a two-stage distribution strategy to realize part attribution of each pixel in the morphological mask and obtaining a part mask; performing motion analysis based on sparse optical flow on each part of the morphological mask of the same moving object according to the part mask to determine the state of the part as a motion state or a static state; and when the state is the motion state, taking the part mask of the corresponding part as a dynamic part mask. The application can realize marking the part mask of the part determined as the motion state as the dynamic part mask.

Description

Technical Field

[0001] This invention relates to an image processing method and system for ORB-SLAM3, belonging to the field of visual image processing and robot navigation technology. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a key technology for robots to achieve autonomous navigation. It allows robots to simultaneously build an environmental map and determine their own position within the map while moving in unknown environments, using onboard sensors (such as cameras and LiDAR). Visual SLAM, which uses cameras as the primary sensor, has been widely researched and applied due to its low cost and rich information content.

[0003] However, most traditional visual SLAM algorithms are based on a core assumption: the scene is static. In the real world, especially in indoor and outdoor environments with frequent human activity, this assumption often does not hold true. Dynamic objects in the scene, such as walking people and moving vehicles, can introduce incorrect visual feature matching, causing the SLAM system to generate incorrect motion estimates. This can range from reducing localization accuracy to, in severe cases, causing tracking loss.

[0004] To address this issue, researchers have proposed various dynamic SLAM methods. One mainstream approach is a passive detection method based on deep learning, such as using networks like YOLO and R-CNN to identify known dynamic categories in the scene (e.g., "people" or "vehicles"), and then masking the entire regions corresponding to these objects during the SLAM process. The advantage of this method is its ability to effectively eliminate most dynamic interference, but its limitations are also significant: it treats dynamic objects as a whole. For example, a person sitting in a chair has most of their body still, with only their arms or head possibly moving slightly. Masking the entire person would result in the loss of a large number of valuable static features on the person (such as clothing texture), which is unacceptable in scenes where dynamic objects occupy most of the field of view.

[0005] Another type of method attempts to detect dynamic regions through pure geometric cues (such as multi-view geometry and optical flow methods). However, these methods are very sensitive to the motion patterns and speeds of objects, and they are not effective when dealing with slowly moving or temporarily stationary objects. In addition, the computational load is large, making it difficult to meet real-time requirements.

[0006] Therefore, existing technologies have significant gaps in accurately distinguishing between the "dynamic parts" and "static parts" of dynamic objects. Achieving this fine-grained dynamic perception would allow SLAM systems to retain more effective visual information while accurately eliminating motion interference, thereby fundamentally improving their performance in real dynamic environments.

[0007] In view of this, the present invention is hereby proposed. Summary of the Invention

[0008] This invention provides an image processing method and system for ORB-SLAM3, which processes the input RGB image in ORB-SLAM3 to obtain dynamic part masks, thereby achieving visual ORB-SLAM3 optimization in dynamic scenes.

[0009] The technical solution of this invention is:

[0010] According to a first aspect of the present invention, an image processing method for ORB-SLAM3 is provided, comprising: inputting an image frame; performing instance segmentation of the RGB image in the input current image frame to obtain a detection result; setting filtering conditions to perform validity filtering and post-processing on the pixel-level mask in the detection result to obtain a morphological mask; wherein the detection result includes bounding boxes, pixel-level masks, confidence scores, and class labels; taking the corresponding bounding boxes of the RGB image in the current image frame that have morphological masks as first bounding boxes; performing ROI cropping and preprocessing on each first bounding box to obtain a preprocessed ROI image; and processing each preprocessed ROI image... The image is sequentially subjected to network inference and inverse coordinate transformation, calculation of auxiliary keypoints, verification of the validity of standard keypoints, and missing completion operations to obtain pose keypoints; the body part skeleton topology is defined; based on the body part skeleton topology, skeletal segments of each part are constructed from the pose keypoints of each preprocessed ROI image; a two-stage allocation strategy is used to assign each pixel in the morphological mask to a part, obtaining the part mask; for each part of the morphological mask of the same moving object, motion analysis based on sparse optical flow is performed based on the part mask to determine whether the part is in motion or at rest; when the state is in motion, the part mask of the corresponding part is used as the dynamic part mask.

[0011] Furthermore, the filtering criteria are as follows: and ;in, The confidence level in the test results. The confidence threshold; bounding boxes in the detection results The pixel area; This is the minimum area threshold.

[0012] Further, the preprocessed ROI images are sequentially subjected to network inference and inverse coordinate transformation, auxiliary keypoint calculation, standard keypoint validity verification, and missing point completion operations to obtain pose keypoints. Specifically, the preprocessed ROI images are subjected to network inference and inverse coordinate transformation to obtain the coordinates and corresponding confidence scores of standard keypoints in the original RGB image coordinate system; based on the coordinates and corresponding confidence scores of standard keypoints in the original RGB image coordinate system, the coordinates and corresponding confidence scores of auxiliary keypoints in the original RGB image coordinate system are calculated; multiple validity verifications are performed on each type of standard keypoint output from each preprocessed ROI image, and the standard keypoints that pass the verification are taken as valid standard keypoints, while those that fail the validity verification are taken as invalid standard keypoints; missing point completion is performed on the standard keypoints among the invalid standard keypoints to obtain completed standard keypoints; the obtained valid standard keypoints, auxiliary keypoints, and completed standard keypoints are taken as pose keypoints.

[0013] Furthermore, the multiple validity verification specifically involves: first, checking whether the confidence level of the standard key point exceeds a first threshold; second, verifying whether the coordinates of the standard key point in the original RGB image coordinate system fall within the corresponding morphological mask; if the confidence level of the standard key point exceeds the first threshold and the coordinates of the standard key point in the original RGB image coordinate system fall within the corresponding morphological mask, then the standard key point is considered to have passed the verification.

[0014] Furthermore, the two-stage allocation strategy for assigning the location of each pixel in the morphological mask to obtain the location mask includes: the first stage pixel allocation: traversing each pixel within the morphological mask corresponding to each preprocessed ROI image. Get pixels The shortest distance to each horizontal and vertical line segment of the head in the body skeleton topology: If the shortest distance is less than the radius of the capsule body of the locked part, then the pixel... Assign to the locked region and obtain the region mask; otherwise, assign the pixel. The first stage is unallocated pixels; the second stage is pixel allocation: the parts in the body part skeleton topology other than the locked parts are taken as the first parts; based on the unallocated pixels in the first stage, competitive allocation is carried out in the first parts.

[0015] Furthermore, the moving object is a human body, and the skeletal topology of the body parts includes the head, torso, left arm, right arm, left leg, and right leg. The head is used as the locking part, and the torso, left arm, right arm, left leg, and right leg are used as the first parts.

[0016] Furthermore, the image processing method for ORB-SLAM3 further includes: using the dynamic part mask as input to the ORB-SLAM3 tracking thread, establishing human pose vertices, human-camera observation constraint edges, human motion prior constraint edges, and skeletal segment length constraint edges for the local mapping thread of ORB-SLAM3, and performing joint optimization solution.

[0017] Further, the competitive allocation specifically involves: obtaining the shortest distance from each unallocated pixel in the first stage to the skeletal segments in each part of the first region; identifying regions where the shortest distance is less than the capsule radius of each part of the first region as competing regions for the current unallocated pixel; calculating the confidence-weighted distance of the skeletal segments of the current unallocated pixel in each competing region; calculating a base score based on the confidence-weighted distance; traversing the competing regions, selecting the region with the lowest base score as a candidate region, and performing a distance threshold check on the candidate region: if the shortest distance from the unallocated pixel to the skeletal segments of the candidate region is less than a preset multiple of the capsule radius of the candidate region, the current unallocated pixel is allocated to the candidate region; otherwise, it remains unallocated; finally, the pixels that pass the distance threshold check are marked as the region mask of the corresponding candidate region.

[0018] Furthermore, for each part of the morphological mask of the same moving object, motion analysis based on sparse optical flow is performed according to the part mask to determine whether the part is in a moving state or a stationary state; when the state is in a moving state, the part mask of the corresponding part is used as a dynamic part mask, including: when the morphological mask of the same moving object exists in the current image frame and the previous image frame, for each part of the morphological mask of the same moving object, the union of the part masks of the corresponding parts in the current image frame and the previous image frame is calculated as the tracking region; within the tracking region, the intersection of the part masks of the corresponding parts in the two frames is calculated as the effective tracking part mask, and the effective tracking... The corner points of the previous image frame are extracted using a corner detection algorithm within the part mask. A 5-layer pyramid Lucas-Kanade optical flow algorithm is used to track the corner points of the previous image frame to the current image frame. Forward-backward consistency checks are performed on the corner points tracked to the current image frame to obtain valid corner point matching pairs. Based on the valid corner point matching pairs, a random sample consistency algorithm is used to filter out outliers to obtain inliers. Based on the inliers, the motion amplitude is calculated. Based on the relationship between the motion amplitude and the motion determination threshold and the proportion of inliers, the state of the part is determined to be either a moving state or a stationary state. When the state is a moving state, the part mask of the corresponding part is used as the dynamic part mask.

[0019] According to a second aspect of the present invention, an image processing system for ORB-SLAM3 is provided, comprising a module of the image processing method for ORB-SLAM3 described in any of the preceding claims.

[0020] The beneficial effects of this invention are as follows: This invention performs instance segmentation and morphological filtering on the input RGB image, effectively filtering out background noise and detecting edge artifacts, ensuring that the generated mask accurately covers potential moving objects. Furthermore, by extracting bounding boxes and performing ROI cropping and preprocessing, subsequent complex calculations (such as pose estimation) are confined to the region of interest, avoiding the processing of redundant information across the entire image, significantly reducing the system's computational load, and improving real-time processing capabilities. Based on this, by obtaining pose keypoints through network inference and combining missing completion and skeleton topology construction, the pose changes of objects can be understood at the structural level, rather than just pixel-level clustering. Furthermore, by utilizing a two-stage allocation strategy, the morphological mask is accurately mapped to specific body parts (such as arms, legs, and torsos), thus overcoming the problems of blurred mask edges or part adhesion in traditional methods, and obtaining high-precision part-level semantic masks. Furthermore, by performing sparse optical flow-based motion analysis independently on each part, the overall motion state of the object can be decoupled from the local limb motion state (i.e., distinguishing between "the whole body is moving" and "only the hand is moving while the torso is stationary"). Then, only parts identified as moving are marked as dynamic part masks for removal, while retaining the visual information of stationary parts. Ultimately, this part-level refined processing avoids the waste of feature points caused by indiscriminately removing entire parts (such as a stationary torso) due to local movement (such as a pedestrian waving). Thus, while ensuring the elimination of dynamic interference, it retains more high-quality static feature points located in stationary parts of the human body for the ORB-SLAM3 system, significantly improving the pose estimation accuracy and mapping stability of the system in dynamic and complex scenes. Attached Figure Description

[0021] Figure 1 This is a flowchart of the present invention.

[0022] Figure 2 This is a trajectory comparison diagram of ORB-SLAM3 provided in Embodiment 2 of the present invention and the present invention on the fr3_walking_halfsphere dataset; Figure 2 (a) is a trajectory comparison diagram of ORB-SLAM3 on the fr3_walking_halfsphere dataset according to Embodiment 2 of the present invention. Figure 2 (b) is a trajectory comparison diagram of the present invention on the fr3_walking_halfsphere dataset according to Embodiment 2 of the present invention.

[0023] Figure 3 This is a trajectory comparison diagram of ORB-SLAM3 and the fr3_walking_rpy dataset provided in Embodiment 2 of the present invention; Figure 3(a) is a trajectory comparison diagram of ORB-SLAM3 on the fr3_walking_rpy dataset according to Embodiment 2 of the present invention. Figure 3 (b) is a trajectory comparison diagram of the present invention on the fr3_walking_rpy dataset according to Embodiment 2 of the present invention.

[0024] Figure 4 This is a trajectory comparison diagram of ORB-SLAM3 provided in Embodiment 2 of the present invention and the present invention on the fr3_walking_xyz dataset; Figure 4 (a) is a trajectory comparison diagram of ORB-SLAM3 on the fr3_walking_xyz dataset according to Embodiment 2 of the present invention. Figure 4 (b) is a trajectory comparison diagram of the present invention on the fr3_walking_xyz dataset according to Embodiment 2 of the present invention.

[0025] Figure 5 This is a trajectory error curve diagram of ORB-SLAM3 based on Embodiment 2 of the present invention on the fr3_walking_halfsphere dataset; Figure 5 (a) is a trajectory error curve of ORB-SLAM3 on the fr3_walking_halfsphere dataset according to Embodiment 2 of the present invention. Figure 5 (b) is a trajectory error curve of the present invention on the fr3_walking_halfsphere dataset according to Embodiment 2 of the present invention.

[0026] Figure 6 This is based on the ORB-SLAM3 provided in Embodiment 2 of the present invention, and the trajectory error curve of the present invention on the fr3_walking_rpy dataset; Figure 6 (a) is a trajectory error curve of ORB-SLAM3 on the fr3_walking_rpy dataset according to Embodiment 2 of the present invention. Figure 6 (b) is a trajectory error curve of the present invention on the fr3_walking_rpy dataset according to Embodiment 2 of the present invention.

[0027] Figure 7 This is a trajectory error curve diagram of ORB-SLAM3 based on Embodiment 2 of the present invention on the fr3_walking_xyz dataset; Figure 7 (a) is a trajectory error curve of ORB-SLAM3 on the fr3_walking_xyz dataset according to Embodiment 2 of the present invention. Figure 7 (b) is a trajectory error curve of the present invention on the fr3_walking_xyz dataset according to Embodiment 2 of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0029] Example 1: As Figure 1 As shown, according to a first aspect of the present invention, an image processing method for ORB-SLAM3 is provided, comprising: inputting an image frame; performing instance segmentation of the RGB image in the input current image frame to obtain a detection result; setting filtering conditions to perform validity filtering and post-processing on the pixel-level mask in the detection result to obtain a morphological mask; wherein the detection result includes bounding boxes, pixel-level masks, confidence scores, and category labels; taking the corresponding bounding boxes of the RGB image in the current image frame that have morphological masks as first bounding boxes; performing ROI cropping and preprocessing on each first bounding box to obtain a preprocessed ROI image; and processing each preprocessed ROI image. The ROI image is sequentially subjected to network inference and inverse coordinate transformation, calculation of auxiliary keypoints, validation of the validity of standard keypoints, and missing point completion operations to obtain pose keypoints; the body part skeleton topology is defined; based on the body part skeleton topology, skeletal segments of each part are constructed from the pose keypoints of each preprocessed ROI image; a two-stage allocation strategy is used to assign each pixel in the morphological mask to a part, obtaining the part mask; for each part of the morphological mask of the same moving object, motion analysis based on sparse optical flow is performed based on the part mask to determine whether the part is in motion or at rest; when the state is in motion, the part mask of the corresponding part is used as the dynamic part mask.

[0030] Furthermore, the filtering criteria are as follows: and ;in, The confidence level in the test results. The confidence threshold; bounding boxes in the detection results The pixel area; This is the minimum area threshold.

[0031] Further, the preprocessed ROI images are sequentially subjected to network inference and inverse coordinate transformation, auxiliary keypoint calculation, standard keypoint validity verification, and missing point completion operations to obtain pose keypoints. Specifically, the preprocessed ROI images are subjected to network inference and inverse coordinate transformation to obtain the coordinates and corresponding confidence scores of standard keypoints in the original RGB image coordinate system; based on the coordinates and corresponding confidence scores of standard keypoints in the original RGB image coordinate system, the coordinates and corresponding confidence scores of auxiliary keypoints in the original RGB image coordinate system are calculated; multiple validity verifications are performed on each type of standard keypoint output from each preprocessed ROI image, and the standard keypoints that pass the verification are taken as valid standard keypoints, while those that fail the validity verification are taken as invalid standard keypoints; missing point completion is performed on the standard keypoints among the invalid standard keypoints to obtain completed standard keypoints; the obtained valid standard keypoints, auxiliary keypoints, and completed standard keypoints are taken as pose keypoints.

[0032] Furthermore, the multiple validity verification specifically involves: first, checking whether the confidence level of the standard key point exceeds a first threshold; second, verifying whether the coordinates of the standard key point in the original RGB image coordinate system fall within the corresponding morphological mask; if the confidence level of the standard key point exceeds the first threshold and the coordinates of the standard key point in the original RGB image coordinate system fall within the corresponding morphological mask, then the standard key point is considered to have passed the verification.

[0033] Furthermore, the two-stage allocation strategy for assigning the location of each pixel in the morphological mask to obtain the location mask includes: the first stage pixel allocation: traversing each pixel within the morphological mask corresponding to each preprocessed ROI image. Get pixels The shortest distance to each horizontal and vertical line segment of the head in the body skeleton topology: If the shortest distance is less than the radius of the capsule body of the locked part, then the pixel... Assign to the locked region and obtain the region mask; otherwise, assign the pixel. The first stage is unallocated pixels; the second stage is pixel allocation: the parts in the body part skeleton topology other than the locked parts are taken as the first parts; based on the unallocated pixels in the first stage, competitive allocation is carried out in the first parts.

[0034] Furthermore, the moving object is a human body, and the skeletal topology of the body parts includes the head, torso, left arm, right arm, left leg, and right leg. The head is used as the locking part, and the torso, left arm, right arm, left leg, and right leg are used as the first parts.

[0035] Furthermore, the image processing method for ORB-SLAM3 further includes: using the dynamic part mask as input to the ORB-SLAM3 tracking thread, establishing human pose vertices, human-camera observation constraint edges, human motion prior constraint edges, and skeletal segment length constraint edges for the local mapping thread of ORB-SLAM3, and performing joint optimization solution.

[0036] Further, the competitive allocation specifically involves: obtaining the shortest distance from each unallocated pixel in the first stage to the skeletal segments in each part of the first region; identifying regions where the shortest distance is less than the capsule radius of each part of the first region as competing regions for the current unallocated pixel; calculating the confidence-weighted distance of the skeletal segments of the current unallocated pixel in each competing region; calculating a base score based on the confidence-weighted distance; traversing the competing regions, selecting the region with the lowest base score as a candidate region, and performing a distance threshold check on the candidate region: if the shortest distance from the unallocated pixel to the skeletal segments of the candidate region is less than a preset multiple of the capsule radius of the candidate region, the current unallocated pixel is allocated to the candidate region; otherwise, it remains unallocated; finally, the pixels that pass the distance threshold check are marked as the region mask of the corresponding candidate region.

[0037] Furthermore, for each part of the morphological mask of the same moving object, motion analysis based on sparse optical flow is performed according to the part mask to determine whether the part is in a moving state or a stationary state; when the state is in a moving state, the part mask of the corresponding part is used as a dynamic part mask, including: when the morphological mask of the same moving object exists in the current image frame and the previous image frame, for each part of the morphological mask of the same moving object, the union of the part masks of the corresponding parts in the current image frame and the previous image frame is calculated as the tracking region; within the tracking region, the intersection of the part masks of the corresponding parts in the two frames is calculated as the effective tracking part mask, and the effective tracking... The corner points of the previous image frame are extracted using a corner detection algorithm within the part mask. A 5-layer pyramid Lucas-Kanade optical flow algorithm is used to track the corner points of the previous image frame to the current image frame. Forward-backward consistency checks are performed on the corner points tracked to the current image frame to obtain valid corner point matching pairs. Based on the valid corner point matching pairs, a random sample consistency algorithm is used to filter out outliers to obtain inliers. Based on the inliers, the motion amplitude is calculated. Based on the relationship between the motion amplitude and the motion determination threshold and the proportion of inliers, the state of the part is determined to be either a moving state or a stationary state. When the state is a moving state, the part mask of the corresponding part is used as the dynamic part mask.

[0038] According to a second aspect of the present invention, an image processing system for ORB-SLAM3 is provided, comprising a module of the image processing method for ORB-SLAM3 described in any of the preceding embodiments.

[0039] Example 2: The method of this invention is built upon the open-source ORB-SLAM3 system framework with functional extensions and in-depth improvements. Its core innovation lies in the collaborative completion of multiple steps (i.e., steps two through five, and step seven below), which are invoked within the ORB-SLAM3 system to achieve refined processing of each frame of the image. The following is the execution process in the ORB-SLAM3 system after incorporating the method of this invention:

[0040] Step 1: ORB-SLAM3 system initialization. During the system startup phase, the main control unit completes the initialization of standard modules such as the original ORB-SLAM3 tracking thread (Tracker), local mapping thread (Local Mapper), loop closing thread (Loop Closer), and ORB bag of words.

[0041] Using the method of the present invention as a preliminary step in tracing threads includes the following:

[0042] Step 2: Input image frames (each frame contains an RGB image and a depth image); feed the RGB image from the current image frame into the trained YOLOv8-Seg instance segmentation model to perform instance segmentation of moving objects, and output the detection results. ,in For bounding box, , The coordinates of the top-left corner of the bounding box. , The coordinates of the bottom right corner of the bounding box. A pixel-level mask. For confidence level, The system assigns category labels (in this embodiment, the moving object is a person, and the category label is a person); sets filtering conditions to filter the effectiveness of pixel-level masks in the detection results, and uses the pixel-level masks that meet the filtering conditions as the filtered pixel-level masks; when a filtered pixel-level mask exists, it performs post-processing operations on the filtered pixel-level mask (the post-processing operations are to perform morphological opening and closing operations on the filtered pixel-level mask to remove noise and fill holes) to obtain a morphological mask; otherwise, it directly uses the original image frame as the input of the original tracking thread in the ORB-SLAM3 system.

[0043] The filtering criteria are as follows: and ;in, The confidence threshold is set to 0.5. The minimum area threshold is set to 1000 pixels. bounding box The pixel area; that is, the confidence level corresponding to the pixel-level mask in the detection result. and If the current pixel-level mask is obtained, then the current pixel-level mask is the required filtered pixel-level mask. It should be noted that if the filtered pixel-level mask is not obtained after filtering according to the filtering conditions, the current image frame is directly sent to the original tracking thread.

[0044] Step 3: Pose key point detection, which includes five sub-processes: ROI pruning and preprocessing, network inference and inverse coordinate transformation, calculation of auxiliary key points, validation of the validity of standard key points, and missing point completion.

[0045] ROI Cropping and Preprocessing Sub-process: The corresponding bounding boxes with morphological masks in the RGB image of the current image frame are taken as the first bounding boxes; to avoid the loss of key points at the boundary, each first bounding box is expanded to obtain the expanded bounding boxes; based on the expanded bounding boxes, the RGB image is cropped to obtain the region of interest (ROI); the cropped ROI is scaled to the standard input size of the YOLOv8-Pose network (e.g., 640×640 pixels) through bilinear interpolation to obtain the preprocessed ROI image, and the scaling transformation matrix is ​​recorded for subsequent inverse coordinate transformation.

[0046] The expanded bounding box expression is:

[0047] ;

[0048] in, This is the expanded bounding box, i.e., the region of interest; The bounding box before expansion Width and height, This is the expansion coefficient.

[0049] Network inference and inverse coordinate transformation: The preprocessed ROI images are fed into the trained YOLOv8-Pose pose estimation network, which outputs the local coordinates of the detected standard keypoints. and the corresponding confidence level ,in , , For the first Confidence of class-standard key points For the first Local coordinates of standard keypoints (for example, if 10 types of standard keypoints are detected, then...) It should be noted that if no standard keypoint is detected, no output will be provided. The 17 standard keypoint categories are as follows: {NOSE(0), LEFT_EYE(1), RIGHT_EYE(2), LEFT_EAR(3), RIGHT_EAR(4), LEFT_SHOULDER(5), RIGHT_SHOULDER(6), LEFT_ELBOW(7), RIGHT_ELBOW(8), LEFT_WRIST(9), RIGHT_WRIST(10), LEFT_HIP(11), RIGHT_HIP(12), LEFT_KNEE(13), RIGHT_KNEE(14), L EFT_ANKLE(15), RIGHT_ANKLE(16)}; NOSE represents the nose, LEFT represents the left, RIGHT represents the right, EYE represents the eye, EAR represents the ear, SHOULDER represents the shoulder, ELBOW represents the elbow, WRIST represents the wrist, HIP represents the hip, KNEE represents the knee, and ANKLE represents the ankle; LEFT_EYE(1) represents the standard key point of the left eye, with a category number of 1, and the others are similar; LEFT_EYE(1) and RIGHT_EYE(2) are mirror images, LEFT_EAR(3) and RIGHT_EAR(4) are mirror images, and the others are similar.

[0050] The local coordinates of standard key points are mapped back to the original RGB image coordinate system through inverse coordinate transformation:

[0051] ;

[0052] in, , These are the coordinates of the standard key points in the original RGB image coordinate system; This refers to the scaling ratio; , These are horizontal and vertical offsets, respectively. The width and height of the original RGB image.

[0053] Calculate auxiliary keypoints: Based on the coordinates and corresponding confidence scores of the standard keypoints in the original RGB image coordinate system, calculate the coordinates and corresponding confidence scores of the auxiliary keypoints in the original RGB image coordinate system.

[0054] The calculation of the coordinates and corresponding confidence levels of auxiliary keypoints in the original RGB image coordinate system based on the coordinates and corresponding confidence levels of standard keypoints in the original RGB image coordinate system is as follows: when there are shoulder standard keypoints among the output standard keypoints, the coordinates of the neck auxiliary keypoint NECK(17) in the original RGB image coordinate system are calculated based on the coordinates of the midpoint of the shoulder standard keypoints in the original RGB image coordinate system. and the corresponding confidence level When there are bi-hip standard key points in the output standard key points, the coordinates of the hip midpoint auxiliary key point MID_HIP(18) in the original RGB image coordinate system are calculated based on the coordinates of the midpoint of the bi-hip standard key points in the original RGB image coordinate system. and the corresponding confidence level .

[0055] Standard key point validity verification: Perform multiple validity verifications on each type of standard key point output from each preprocessed ROI image. Standard key points that pass the verification are considered valid standard key points, while standard key points that fail the validity verification are considered invalid standard key points.

[0056] First, check whether the confidence level of the standard key points exceeds the first threshold. Secondly, verify whether the coordinates of the standard keypoints in the original RGB image coordinate system fall within the corresponding morphological mask; that is, as mentioned above, if the confidence level of the standard keypoints exceeds the first threshold, then the condition is met. Furthermore, the coordinates of the standard keypoints in the original RGB image coordinate system fall within the corresponding morphological mask, i.e. , If a standard key point is located within the region of the morphological mask in the original RGB image coordinate system, it is considered to have passed the verification. The verified standard key points are considered to be valid standard key points, and the standard key points that fail the validity verification are considered to be invalid standard key points.

[0057] Missing keypoints: For each keypoint among invalid keypoints, if a matching mirror image exists among valid keypoints, the corresponding keypoints among the invalid keypoints with mirror images are inferred using the symmetry of the human skeleton to achieve missing keypoint completion, and these are used as completion keypoints. If the current invalid keypoint has no matching mirror image among valid keypoints, it is removed. The obtained valid keypoints, auxiliary keypoints, and completion keypoints are used as pose keypoints.

[0058] For example: Let the trunk midline be the line connecting the midpoints of the left and right shoulders and the midpoints of the left and right hips. For missing standard key points on one side of the limb, mirror inference is performed based on the mirror position of the standard key points that have passed the validity verification on the contralateral side about the midline. The mirror inference formula is:

[0059] ;

[0060] in, Point On the central axis The projection on The key criteria for passing the validity verification of the opposite side are as follows: This represents the completed standard keypoints obtained after mirror inference. The confidence level of each standard keypoint in the completed standard keypoints is set to half the confidence level of the standard keypoints that passed the validity verification on the other side before mirror inference, in order to reflect their uncertainty.

[0061] Step 4: Define the skeleton topology of body parts; based on the skeleton topology of body parts, construct the skeletal segments of each part in each preprocessed ROI image; use a two-stage allocation strategy to assign the part to each pixel in the morphological mask and obtain the part mask.

[0062] Step four includes:

[0063] Define the skeletal topology of the body parts, dividing the body into 6 parts: head, trunk, left arm, right arm, left leg, and right leg;

[0064] The skeletal topology of the body parts includes:

[0065] HEAD: Contains 10 skeletal segments, representing the horizontal and vertical segments of five standard keypoints: NOSE, LEFT_EYE, RIGHT_EYE, LEFT_EAR, and RIGHT_EAR. The horizontal and vertical segments form a cross shape. The horizontal segments start from (… )arrive( ), the vertical line segment from ( )arrive( );in, Standard key points in the HEAD region Coordinates in the original RGB image coordinate system, radius of the head capsule , The width of the expanded bounding box is defined by the above-mentioned standard key points corresponding to the header. Horizontal and vertical line segments are constructed respectively.

[0066] TORSO (Torso): Contains 5 skeletal segments. These are the central axis... The central axis is the line connecting the two auxiliary keypoints NECK and MID_HIP; the shoulder width line L_shoulder is the line connecting the two standard keypoints LEFT_SHOULDER and RIGHT_SHOULDER; the hip width line L_hip is the line connecting the two standard keypoints LEFT_HIP and RIGHT_HIP; the left diagonal line L_left is the line connecting the two standard keypoints LEFT_SHOULDER and LEFT_HIP; the right diagonal line L_right is the line connecting the two standard keypoints RIGHT_SHOULDER and RIGHT_HIP. These five skeletal segments form an "X"-shaped support framework. (Tortoise capsule radius) .

[0067] LEFT_ARM (Left Arm): Contains two skeletal segments: the upper left arm segment and the lower left arm segment. The upper left arm segment connects the standard keypoints LEFT_SHOULDER and LEFT_ELBOW, while the lower left arm segment connects the standard keypoints LEFT_ELBOW and LEFT_WRIST. (Left arm capsule radius) .

[0068] RIGHT_ARM (Right Arm): Contains two skeletal segments: the upper right arm line and the lower right arm line. The upper right arm line is the line connecting the two standard keyframes RIGHT_SHOULDER and RIGHT_ELBOW, and the lower right arm line is the line connecting the two standard keyframes RIGHT_ELBOW and RIGHT_WRIST. The radius of the right arm capsule is... .

[0069] LEFT_LEG (Left Leg): Contains two skeletal segments: the upper left leg line and the lower left leg line. The upper left leg line is the line connecting the two standard keypoints LEFT_HIP and LEFT_KNEE, and the lower left leg line is the line connecting the two standard keypoints LEFT_KNEE and LEFT_ANKLE. Left leg capsule radius. , This is the length of the diagonal of the expanded bounding box.

[0070] RIGHT_LEG (Right Leg): Contains two skeletal lines: the upper right leg line and the lower right leg line. The upper right leg line is the line connecting the two standard keypoints RIGHT_HIP and RIGHT_KNEE, and the lower right leg line is the line connecting the two standard keypoints RIGHT_KNEE and RIGHT_ANKLE. Right leg capsule radius. .

[0071] Based on the skeletal topology of the body parts, skeletal segments are constructed for each part based on the valid standard key points, auxiliary standard key points, and complete standard key points obtained from each preprocessed ROI image. That is, skeletal segments are constructed for the "valid standard key points, auxiliary standard key points, and complete standard key points" existing in the current preprocessed ROI image based on the skeletal topology of the body parts, and skeletal segments corresponding to points that do not exist are not included in the construction.

[0072] First stage pixel allocation: Traverse each pixel within the morphological mask corresponding to each preprocessed ROI image. , obtain The shortest distance to each horizontal and vertical line segment of the head in the skeletal topology of the body part. If the shortest distance is less than the radius of the head capsule, then... Then the pixels Assign to the HEAD region to obtain the head region mask; otherwise, assign pixels. As unassigned pixels; the HEAD region is the locked region.

[0073] For example: pixel The coordinates in the original RGB image coordinate system are ( The starting point of the horizontal or vertical line segment in HEAD is... The destination is ,distance for:

[0074] ;

[0075] There are a total of 10 horizontal or vertical line segments in HEAD. Based on the above formula, 10 distances can be obtained. The minimum value among the 10 distances is taken as... .

[0076] Second-stage pixel allocation:

[0077] Five regions, excluding the head, are designated as the first region. Based on the unallocated pixels from the first stage, competitive allocation is performed within the first region (i.e., in TORSO, LEFT_ARM, RIGHT_ARM, LEFT_LEG, and RIGHT_LEG). Specifically, this competitive allocation involves first obtaining the shortest distance from each unallocated pixel from the first stage to the skeletal line segment within each region of the first region. The calculation method is the same as above, and the statistics are... The region is designated as the competing region for currently unallocated pixels; among which, for , , , or (If the calculated shortest distance is the shortest distance from the unassigned pixel to the bone segment in the TORSO region, then) for (The same applies to others).

[0078] Calculate the confidence-weighted distance of the skeletal segments of the currently unassigned pixels in each competing region. The expression is:

[0079] ;

[0080] in, Unallocated pixels The shortest distance to the bone segment in the competing part. The confidence score of a skeletal segment in a competing region is determined by taking the smaller of the confidence scores of the two endpoints of the skeletal segment.

[0081] The base score is calculated based on confidence-weighted distance. ;in The distance is weighted by confidence level. The radius of the capsule at the competing site. Priority weights for competing body parts (priority weights for the torso) Priority weights for left and right arms Priority weights for left and right legs ).

[0082] Traverse the competing regions, select the region with the lowest base score as the candidate region, and perform a distance threshold check on the candidate region: if the shortest distance from the unassigned pixel to the skeletal line segment in the candidate region is less than 4 times the capsule radius of the candidate region, assign the currently unassigned pixel to the candidate region; otherwise, keep it unassigned; finally, mark the pixels that pass the distance threshold check as the region mask of the corresponding candidate region, and thus obtain the region mask of the first region.

[0083] For example, with unassigned pixels For example, suppose The shortest distance to the skeletal segment in the TORSO region is ( ), The shortest distance to the bone segment in the LEFT_ARM region is ( ), The shortest distance to the bone segment in the RIGHT_ARM region is ( ), The shortest distance to the skeletal line segment in the LEFT_LEG region is ( ), The shortest distance to the bone segment in the RIGHT_LEG region is ( Based on the above, statistics , The regions defined as competition regions, namely the LEFT_LEG region and the RIGHT_LEG region, are considered as unallocated pixels. The competing regions are identified. The confidence-weighted distance of the currently unassigned pixel to the skeletal segments within the competing regions is calculated to obtain the unassigned pixel. Confidence-weighted distance of the upper left leg line in the LEFT_LEG region Unallocated pixels Confidence-weighted distance of the lower left leg line in the LEFT_LEG region Unallocated pixels Confidence-weighted distance of the upper right leg line in the RIGHT_LEG region Unallocated pixels Confidence-weighted distance of the lower right leg line in the RIGHT_LEG region This yields four basic scores; assuming the region with the lowest basic score is the right leg, then the right leg is the candidate region and no pixels are assigned to it. The shortest distance to the skeletal segment in the right leg is less than four times the radius of the right leg capsule, therefore no pixels are allocated. It was assigned to the right leg.

[0084] Step 5: For each part of the morphological mask of the same moving object, perform motion analysis based on sparse optical flow according to the part mask to determine whether the part is in motion or at rest; when the state is in motion, use the part mask of the corresponding part as the dynamic part mask.

[0085] Step five includes:

[0086] When the morphological mask of the same moving object exists in the current image frame and the previous image frame, for each part of the morphological mask of the same moving object, the union of the part masks of the corresponding parts in the current image frame and the previous image frame is calculated as the tracking region R; within the tracking region R, the intersection of the part masks of the corresponding parts in the two frames is calculated as the effective tracking part mask. Masking in the effective tracking area The Shi-Tomasi corner detection algorithm is used to extract the corners of the previous image frame (the quality threshold in the Shi-Tomasi algorithm is 0.01, and the minimum spacing is 10 pixels).

[0087] A 5-layer pyramid Lucas-Kanade optical flow algorithm is used to track the corner points of the previous image frame to the current image frame (search window 41×41 pixels, maximum iteration 30 times).

[0088] Perform forward-backward consistency check on the corner points tracked to the current image frame to obtain valid corner point matching pairs: track the corner points of the current image frame obtained by forward tracking back to the previous frame, calculate the round-trip error, remove corner point matching pairs with round-trip error exceeding 1.5 pixels, and take the remaining corner points as valid corner points.

[0089] Based on the effective corner point matching pairs, the Random Sample Consensus Algorithm (RANSAC) is used to filter out outliers and obtain inliers: three pairs of matching points are randomly sampled to calculate the affine transformation, and inliers with reprojection errors of less than 3 pixels are counted. The model with the most inliers is retained.

[0090] Calculate the amplitude of motion based on the interior points: ;in, The number of interior points, For image frames The Middle One internal point;

[0091] Based on the relationship between motion amplitude and motion determination threshold, and the proportion of inliers, the state of a part is determined to be either in motion or stationary. When the state is in motion, the part mask of the corresponding part is used as the dynamic part mask. Specifically: if m > τ_motion (τ_motion is the motion determination threshold) and the proportion of inliers > 50%, the corresponding part is determined to be in motion; otherwise, it is determined to be stationary. If determined to be in motion, the part mask of the corresponding part is used as the dynamic part mask.

[0092] Step Six: The dynamic part mask is used as input to the original tracking thread of ORB-SLAM3, i.e., the generated dynamic part mask is passed to the ORB feature extractor object. The ORB feature extractor first uses the FAST corner detection algorithm to detect candidate corners across the entire image, forming an initial set of candidate feature points. Then, it checks the pixel coordinates (x, y) of each candidate feature point in the initial set to determine if it falls within any dynamic part mask region: if the pixel value at (x, y) of the dynamic part mask is 255 (dynamic region), the candidate feature point is marked as invalid and discarded; if the pixel value is 0 (static region), the candidate feature point is retained for subsequent steps in the tracking thread (i.e., ...). Figure 1 The process includes map initialization, constant velocity model, local map, relocalization, keyframe decision-making (the tracking thread is the existing well-known ORB-SLAM3, which is not described in detail here), and obtaining keyframes.

[0093] Step 7: When the moving object is a human body, improve the local BA optimization in the local mapping thread. The improvement is as follows:

[0094] Establish human pose vertices, human-camera observation constraint edges, human motion prior constraint edges, and skeletal segment length constraint edges, and perform joint optimization to solve the problem.

[0095] The human pose vertices are used to store and represent the six-DOF human pose of the hip midpoint auxiliary keypoint in the world coordinate system; the six-DOF human pose data structure adopts a Lie group. Manifold form representation, specifically using quaternions to represent rotational attitudes. The three-dimensional translation vector o representing the spatial position is parameterized as follows: During the BA optimization process, the six-DOF human pose data is incrementally updated by mapping to Lie algebras. Meanwhile, to reduce the dimensionality of the optimization variables and ensure trajectory smoothness, a time quantization strategy is adopted to manage the human pose vertices: the system time axis is divided into continuous time intervals of 0.1 seconds, and all visual observation data (pose keypoint coordinates) of the same human body falling within the same time interval share the same human pose vertex.

[0096] The human-camera observation constraint edge is connected to the camera pose vertex. (Representing the camera's pose in the world coordinate system) and the vertex of the human pose The binary edges are used to constrain the relative pose geometry between the camera and the human body; the relative pose of the auxiliary keypoint at the midpoint of the human hip in the camera coordinate system is calculated by combining the effective standard keypoint detection in step three with the depth image. (Representing the human body's pose measurement relative to the camera) is used as the measured pose. A human-camera observation error function is constructed to minimize the human body pose vertex. The difference between the measured pose and the Lie algebra space is expressed as follows: ;in, For the logarithmic mapping from the Lie group SE(3) to the Lie algebra, This represents the inverse transformation of the human body's pose vertex. To mitigate the impact of antisymmetric matrix manipulation, the human-camera observation error obtained from the human-camera observation error function is used to correct the human and camera poses during the BA optimization process. Simultaneously, to improve system robustness, a dynamic weighting strategy is employed: the information matrix weights of the human-camera observation constraint edges are dynamically calculated based on the number of pose keypoints and the average confidence level in the current image frame. The corresponding human-camera observation constraint edges are only added to the factor graph when the number of pose keypoints is greater than or equal to a first preset threshold.

[0097] The relative pose (Representing the pose measurement of the human body relative to the camera) is used as the measurement pose. This is achieved by analyzing each valid standard keypoint in each preprocessed ROI image. , The depth value Z of the sampled depth image is obtained and back-projected into 3D space according to the camera intrinsic parameters to obtain the effective standard keypoint coordinates in the camera coordinate system (i.e., the position information in the pose measurement values ​​of the human body relative to the camera). The calculation formula is as follows: ;in, , The coordinates of the camera's optical center pixel. , For camera focal length, The coordinates of the effective standard key points in the camera coordinate system are used; further, the pose information of the human body relative to the camera is obtained based on the effective standard key point coordinates in the camera coordinate system.

[0098] The prior constraint edges for human motion are the vertices connecting the poses of the same human body at adjacent moments. (the peak of the human body pose at time t) and A binary edge (the human pose vertex at time t+1) is used to constrain pose changes between adjacent time steps based on the assumption of continuity of human motion. By calculating the relative transformation between poses at two adjacent time steps as motion measurements, a motion error function is constructed to penalize sudden changes in human pose within a short time, achieving smooth constraint on the motion trajectory. The expression for the human motion prior error function is as follows: During the BA optimization process, the prior error of human motion obtained by the prior error function of human motion forces the update of human pose to conform to the laws of physical motion. At the same time, in order to adapt to motion changes at different frame rates, an adaptive covariance adjustment strategy is adopted: the weight of the information matrix of the human motion prior constraint edge is adaptively adjusted according to the time interval between adjacent vertices. The longer the time interval, the weaker the constraint strength. The Huber robust kernel function is introduced to reduce the interference caused by sudden motion.

[0099] The skeletal segment length constraint edge connects to a single human pose vertex. The unary edge is used to limit scale drift by utilizing the rigidity of human anatomical structures. First, a standard skeleton model is introduced, which predefines the relative positions of the joints in the human body's local coordinate system. This local coordinate system is established with the midpoint of the hip as the origin and the line connecting the left and right hips as the horizontal axis, and is independent of the absolute position of the human body in the world. The coordinates of the joints at both ends of a skeletal segment (i.e., the starting and ending pose keypoints of a rigid bone defined in the skeleton topology) in the standard skeleton model are called the joint local coordinates. By transforming the local coordinates of the joints to the world coordinate system through the human pose to be optimized, the difference between the transformed Euclidean distance between the two joints and the desired length is calculated as the geometric measurement error. A geometric measurement error function for skeletal segments is constructed to ensure the rationality of the physiological structure of the human skeleton during movement. The expression for the geometric measurement error function for skeletal segments is as follows: ;in Let Euclidean norm be the vector. This represents the expected length of the skeletal segment. During the BA optimization process, the geometric measurement error of the skeletal segment, obtained using the geometric measurement error function, constrains the scale of the human pose. Simultaneously, to adapt to individual body size differences and observation noise, an online adaptive update strategy is adopted: Expected length The ORB-SLAM3 system maintains the data using historical observation data, which refers to the sample sequence of skeletal segment lengths measured by the system in all past frames of the human target. The historical data is filtered by the exponential moving average (EMA) algorithm to obtain a stable length estimate. The corresponding skeletal segment length constraint edge is added to the factor graph only when the detection confidence of the joints at both ends of the skeletal segment is greater than or equal to the second preset threshold.

[0100] The joint optimization solution is the final computational step in BA optimization, used to integrate camera pose, human pose, and map point coordinates to obtain a globally consistent optimal state estimate. By integrating all the constraint edges constructed above, a weighted least squares overall objective function is established, which includes map point reprojection error, human-camera observation error, human motion prior error, and skeletal segment geometric measurement error. The expression is:

[0101] ;

[0102] in, This refers to map point reprojection error. These are, respectively, human-camera observation error, human motion prior error, and skeletal segment length error; The information matrices (inverses of the covariance matrix) corresponding to each error term are used to represent the confidence weights of each measurement. A Huber robust kernel function is used to suppress the impact of mismatches and dynamic outliers on the optimization results. During the BA optimization process, the Levenberg-Marquardt (LM) algorithm is used to iteratively solve the overall objective function. At the current state estimate, the overall objective function is linearized using a first-order Taylor expansion and a normal equation is constructed. The state increment in the Lie algebra space is solved, and the vertex pose is updated using an exponential mapping until the error converges. This completes the joint optimization of camera pose, map point coordinates, and human pose. The optimized camera pose, map point coordinates, and human pose are then used for map construction and loop closure detection via a closed-loop thread.

[0103] Furthermore, the PC was equipped with an Intel i5-12600KF processor, 32GB of RAM, an NVIDIA GeForce RTX 4060 graphics card, and running Ubuntu 18.04 LTS. To verify the method proposed in this invention, the publicly available TUM dataset was used for validation, and the root mean square error (RMSE) of the absolute trajectory error (ATE) was used for quantitative analysis. The TUMRGB-D dataset, provided by the Technical University of Munich, contains RGB images, depth images, and corresponding camera poses captured by the Kinect sensor. This dataset is specifically designed for evaluating and benchmarking visual SLAM algorithms.

[0104] Figure 2 The algorithms compared include: the benchmark algorithm ORB-SLAM3, the excellent algorithm DynaSLAM for dynamic scenes, the algorithm proposed in this invention that only includes the front end (OURS front end, i.e., using steps two to five proposed in this invention, with step seven replaced by the traditional one), and the complete algorithm proposed in this invention (OURS, i.e., using steps two to five and step seven proposed in this invention). The error comparison of different methods is shown in Table 1. Table 1 includes the errors of the sequences fr3_sitting_halfsphere, fr3_sitting_rpy, and fr3_sitting_static for static objects and fr3_walking_halfsphere, fr3_walking_rpy, and fr3_walking_xyz for violently moving objects. As can be seen from Table 1, the error value of the benchmark algorithm ORB-SLAM3 is extremely high; for example, the error in the fr3_walking_rpy sequence is as high as 0.689805m, indicating that it is severely affected by dynamic interference. In contrast, the errors of the OURS front end and OURS proposed in this invention remain at extremely low levels on these sequences, with a significant reduction in error, demonstrating the robustness of the algorithm in dynamic environments. The OURS front-end and OURS of this invention are comparable to the DynaSLAM algorithm specifically designed for dynamic scenes, verifying the superior accuracy of the method presented in this invention. Comparison of the data from "OURS (front-end)" and "OURS" shows that the error of the complete algorithm is smaller than that of the algorithm with only an improved front-end in all test sequences. This intuitively demonstrates that the further optimization strategy implemented in the back-end of this invention is effective, further correcting the estimation bias of the front-end and thus achieving globally optimal positioning accuracy.

[0105] Table 1. Comparison of Errors of Different Methods

[0106]

[0107] Figures 2-4 This paper presents a comparison of trajectory estimation results between the proposed algorithm (OURS) and the baseline algorithm (ORB-SLAM3) on dynamic sequences of the TUM RGB-D dataset. The comparison reveals a significant separation between the estimated trajectory and the actual trajectory in dynamic scenes (such as the fr3_walking_halfsphere, fr3_walking_rpy, and fr3_walking_xyz sequences). This indicates that the baseline algorithm experiences substantial localization drift and fails to closely follow the actual path under interference from moving objects such as pedestrians. In contrast, the proposed algorithm (OURS) achieves a high degree of overlap between the estimated and actual trajectories in the same test sequences.

[0108] Figures 5-7 This further demonstrates a quantitative comparison of the absolute trajectory translation error over time between the algorithm of this invention and the benchmark algorithm. (Observation) Figure 5 (a) Figure 6 (a) Figure 7 (a) Regarding the error curves of ORB-SLAM3, it can be observed that in dynamic scenes, the curves exhibit severe oscillations and significantly higher peak values. For example, in the fr3_walking_xyz sequence, the error curve shows several sharp increases, with peak values ​​even exceeding 0.7 meters, indicating that the algorithm's localization fails severely when dynamic objects appear. In contrast, Figure 5 (b) Figure 6 (b) Figure 7 (b) The error curve of the algorithm (OURS) of this invention remains within a low amplitude range throughout the entire time series, with a generally stable trend. For most of the time period, the error value remains below 0.05 meters. This comparison intuitively confirms that the algorithm of this invention can reduce positioning errors to the centimeter level, significantly improving the positioning accuracy and stability of the system in dynamic environments.

[0109] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. An image processing method for ORB-SLAM3, characterized in that, include: Input image frame; Perform instance segmentation of moving objects in the RGB image of the current input image frame to obtain the detection results; Filtering criteria are set to perform validity filtering and post-processing on the pixel-level masks in the detection results to obtain morphological masks; the detection results include bounding boxes, pixel-level masks, confidence scores, and category labels; The bounding boxes corresponding to the morphological masks in the RGB images of the current image frame are taken as the first bounding boxes; each first bounding box is cropped and preprocessed to obtain a preprocessed ROI image; each preprocessed ROI image is sequentially subjected to network inference and inverse coordinate transformation, calculation of auxiliary key points, verification of the validity of standard key points and missing completion operations to obtain pose key points; Define the skeleton topology of body parts; based on the skeleton topology of body parts, construct the skeletal segments of each part for the pose key points of each preprocessed ROI image; adopt a two-stage allocation strategy to realize the part assignment of each pixel in the morphological mask and obtain the part mask. For each part of the morphological mask of the same moving object, motion analysis based on sparse optical flow is performed according to the part mask to determine whether the part is in motion or at rest; when the state is in motion, the part mask of the corresponding part is used as the dynamic part mask. For each part of the morphological mask of the same moving object, motion analysis based on sparse optical flow is performed according to the part mask to determine whether the part is in a moving state or a stationary state; when the state is in a moving state, the part mask of the corresponding part is used as a dynamic part mask, including: When the current image frame and the previous image frame have the same morphological mask of the same moving object, for each part of the morphological mask of the same moving object, the union of the part masks of the corresponding parts of the current image frame and the previous image frame is calculated as the tracking region; within the tracking region, the intersection of the part masks of the corresponding parts of the two frames is calculated as the effective tracking part mask; within the effective tracking part mask, the corner point detection algorithm is used to extract the corner points of the previous image frame. A 5-layer pyramid Lucas-Kanade optical flow algorithm is used to track the corner points of the previous image frame to the current image frame. Perform a forward-backward consistency check on the corner points tracked up to the current image frame to obtain valid corner point matching pairs; Based on the effective corner point matching pair, the random sample consensus algorithm is used to filter out outliers and obtain inliers; based on the inliers, the motion amplitude is calculated. Based on the relationship between motion amplitude and motion determination threshold and the ratio of interior points, the state of a part is determined to be either in motion or at rest; when the state is in motion, the part mask of the corresponding part is used as the dynamic part mask.

2. The image processing method for ORB-SLAM3 according to claim 1, wherein, The filtering criteria are as follows: and ;in, The confidence level in the test results. The confidence threshold; bounding boxes in the detection results The pixel area; This is the minimum area threshold.

3. The image processing method for ORB-SLAM3 according to claim 1, wherein, Each preprocessed ROI image is sequentially subjected to network inference and inverse coordinate transformation, auxiliary keypoint calculation, standard keypoint validity verification, and missing point completion operations to obtain pose keypoints. Specifically, each preprocessed ROI image is subjected to network inference and inverse coordinate transformation to obtain the coordinates of the standard keypoints in the original RGB image coordinate system and their corresponding confidence scores; based on the coordinates of the standard keypoints in the original RGB image coordinate system and their corresponding confidence scores, the coordinates of the auxiliary keypoints in the original RGB image coordinate system and their corresponding confidence scores are calculated; multiple validity verifications are performed on each type of standard keypoint output from each preprocessed ROI image, and the standard keypoints that pass the verification are taken as valid standard keypoints, while the standard keypoints that fail the validity verification are taken as invalid standard keypoints. For invalid standard key points, missing standard key points are filled in to obtain complete standard key points; The obtained valid standard key points, auxiliary key points, and supplementary standard key points are used as attitude key points.

4. The image processing method for ORB-SLAM3 according to claim 3, wherein, The multiple validity verification specifically involves: first, checking whether the confidence level of the standard key point exceeds the first threshold; and second, verifying whether the coordinates of the standard key point in the original RGB image coordinate system fall within the corresponding morphological mask. If the confidence level of a standard keypoint exceeds the first threshold, and the coordinates of the standard keypoint in the original RGB image coordinate system fall within the corresponding morphological mask, then the standard keypoint is considered to have passed verification.

5. The image processing method for ORB-SLAM3 according to claim 1, wherein, The method employs a two-stage allocation strategy to assign the location of each pixel in the morphological mask and obtain the location mask, including: First stage pixel allocation: Traverse each pixel within the morphological mask corresponding to each preprocessed ROI image. Get pixels The shortest distance to each horizontal and vertical line segment of the head in the body skeleton topology: If the shortest distance is less than the radius of the capsule body of the locked part, then the pixel... Assign to the locked region and obtain the region mask; otherwise, assign the pixel. As unassigned pixels in the first stage; Second-stage pixel allocation: The parts of the body skeleton topology other than the locked parts are taken as the first parts; based on the unallocated pixels in the first stage, competitive allocation is carried out in the first parts.

6. The image processing method for ORB-SLAM3 according to claim 5, wherein, The moving object is a human body, and the skeletal topology of the body parts includes the head, torso, left arm, right arm, left leg, and right leg. The head is used as the locking part, and the torso, left arm, right arm, left leg, and right leg are used as the first parts.

7. The image processing method for ORB-SLAM3 according to claim 6, wherein, The image processing method for ORB-SLAM3 further includes: using the dynamic part mask as the input to the ORB-SLAM3 tracking thread, and establishing human pose vertices, human-camera observation constraint edges, human motion prior constraint edges, and skeletal line segment length constraint edges for the local mapping thread of ORB-SLAM3, and performing joint optimization solution.

8. The image processing method for ORB-SLAM3 according to claim 5, wherein, The competitive allocation is specifically as follows: Obtain the shortest distance from each unallocated pixel in the first stage to the bone line segment in each part of the first part, and count the parts whose shortest distance is less than the capsule radius of each part of the first part as the competing parts of the current unallocated pixel. Calculate the confidence-weighted distance of the skeletal line segments of the currently unassigned pixels in each competing region; The base score is calculated based on confidence-weighted distance; Traverse the competing regions, select the region with the lowest base score as the candidate region, and perform a distance threshold check on the candidate region: if the shortest distance from the unassigned pixel to the skeletal line segment in the candidate region is less than a preset multiple of the capsule radius of the candidate region, assign the currently unassigned pixel to the candidate region; otherwise, keep it unassigned; finally, mark the pixels that pass the distance threshold check as the region mask of the corresponding candidate region.

9. An image processing system for ORB-SLAM3, characterized in that, Includes the module for the image processing method for ORB-SLAM3 as described in any one of claims 1-8.

Images