Vision semantic based positioning and mapping method and device, and electronic equipment

Abstract

Disclosed are a vision semantic-based positioning and mapping method and device and an electronic device. Relate to the technical field of computer vision detection, the method comprises: determining the camera pose of a camera and the depth map corresponding to a target frame image based on the target frame image collected by the camera; determining a target instance in the target frame image and the semantic features of the target instance in the target frame image; determining the depth information of the target instance in the target frame image based on the depth map corresponding to the target frame image and the target instance; and updating a three-dimensional point cloud map based on the camera pose of the camera, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image. The present disclosure can effectively improve the accuracy and robustness of positioning and mapping.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision detection technology, specifically to a method and apparatus for localization and mapping based on visual semantics, and an electronic device. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) typically consists of sensor perception, semantic recognition, instance tracking, and semantic mapping. First, environmental data is acquired using cameras or LiDAR. The semantic recognition module performs category detection and segmentation on objects in the images or point clouds. Then, the instance tracking module matches and associates the same object across different frames to track it. Only when an object is confirmed to be stable and reliable, and its tracking ID shows no drift, is it added to the map. The map is continuously updated and located in real-time as mobile devices (such as robots and vehicles) move.

[0003] In practice, while LiDAR-based SLAM systems can achieve high-precision positioning, they suffer from significant limitations due to their high cost and large size, hindering large-scale widespread application. Furthermore, they are unsuitable for scenarios with strict requirements on device size and cost, thus limiting their practicality. Visual SLAM systems based on other sensors (such as cameras or depth cameras) can effectively reduce costs and device size, but they are susceptible to interference from changes in lighting and the movement of dynamic objects (pedestrians, vehicles, moving objects) in the scene. This makes them prone to tracking loss or ID drift, preventing the construction of robust and reliable high-level semantic maps and restricting the application of SLAM in long-term human-computer interaction, semantic navigation, and other practical scenarios. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides a method, apparatus, and electronic device for localization and mapping based on visual semantics, which can effectively improve the accuracy and robustness of instance tracking in localization and mapping, suppress tracking drift, and achieve continuous and reliable instance tracking.

[0005] The first aspect of this disclosure provides a localization and mapping method based on visual semantics, comprising: determining the camera pose and the depth map corresponding to the target frame image based on the target frame image acquired by the camera; determining the target instance in the target frame image and the semantic features of the target instance in the target frame image; determining the depth information of the target instance in the target frame image based on the depth map corresponding to the target frame image and the target instance; and updating the 3D point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image.

[0006] A second aspect of this disclosure provides a localization and mapping apparatus based on visual semantics, including a processor configured to: determine the camera pose and a depth map corresponding to the target frame image based on a target frame image acquired by a camera; determine a target instance in the target frame image and the semantic features of the target instance in the target frame image; determine the depth information of the target instance in the target frame image based on the depth map corresponding to the target frame image and the target instance; and update a three-dimensional point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image.

[0007] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program that is executed by a processor to perform the visual semantic-based localization and mapping method of the first aspect of this disclosure.

[0008] A fourth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the visual semantic-based localization and mapping method of the first aspect of this disclosure.

[0009] The fifth aspect of this disclosure provides a computer program product that, when executed by an instruction processor, performs the visual semantic-based localization and mapping method proposed in the first aspect of this disclosure.

[0010] The technical solution provided in this disclosure can determine the camera pose and the corresponding depth map of the target frame image based on the target frame image acquired by the camera; then determine the target instance and its semantic features in the target frame image; based on the depth map and the target instance, determine the depth information of the target instance in the target frame image; finally, update the 3D point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image. Therefore, this disclosure can effectively fuse visual semantic information and 3D geometric information by combining camera pose, depth information, and the semantic features of the target instance during the localization and mapping process. It improves the accuracy and robustness of instance tracking in localization and mapping based on the joint constraints of multi-source information, suppresses tracking loss and ID drift, and achieves long-term, continuous, and reliable instance tracking. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a hardware system architecture according to an embodiment of the present disclosure.

[0012] Figure 2This is a flowchart illustrating a visual semantic localization and mapping method provided in an exemplary embodiment of this disclosure.

[0013] Figure 3 This is a schematic diagram of the front-end and back-end processing flow corresponding to the visual semantic localization and mapping method provided in an exemplary embodiment of this disclosure.

[0014] Figure 4 This is a flowchart illustrating a visual semantic localization and mapping method provided in another exemplary embodiment of this disclosure.

[0015] Figure 5 This is a flowchart illustrating a method for locating and mapping visual semantics provided in another exemplary embodiment of this disclosure.

[0016] Figure 6 This is a schematic diagram of the backend processing flow corresponding to the visual semantic localization and mapping method provided in an exemplary embodiment of this disclosure.

[0017] Figure 7 This is a flowchart illustrating a visual semantic localization and mapping method provided in yet another exemplary embodiment of this disclosure.

[0018] Figure 8 This is a flowchart illustrating a visual semantic localization and mapping method provided in yet another exemplary embodiment of this disclosure.

[0019] Figure 9 This is a flowchart illustrating a visual semantic localization and mapping method provided in yet another exemplary embodiment of this disclosure.

[0020] Figure 10 This is a flowchart illustrating a visual semantic localization and mapping method provided in yet another exemplary embodiment of this disclosure.

[0021] Figure 11 This is a flowchart illustrating a visual semantic localization and mapping method provided in yet another exemplary embodiment of this disclosure.

[0022] Figure 12 This is a schematic diagram of the front-end and back-end processing flow corresponding to the visual semantic localization and mapping method provided in another exemplary embodiment of this disclosure.

[0023] Figure 13 This is a schematic diagram of feature extraction and association in a visual semantic localization and mapping method provided by an exemplary embodiment of this disclosure.

[0024] Figure 14 The method for localizing and mapping visual semantics provided in an exemplary embodiment of this disclosure utilizes the combined effects of three types of pose constraints—first relative pose, second relative pose, and third relative pose—to jointly optimize camera pose through a multi-source fusion factor graph architecture.

[0025] Figure 15 This is a flowchart illustrating a visual semantic localization and mapping method provided in yet another exemplary embodiment of this disclosure.

[0026] Figure 16 This is a flowchart illustrating a visual semantic localization and mapping method provided in yet another exemplary embodiment of this disclosure.

[0027] Figure 17 This is a schematic diagram of the front-end and back-end processing flow corresponding to the visual semantic localization and mapping method provided in another exemplary embodiment of this disclosure.

[0028] Figure 18 This is a schematic diagram of the structure of a visual semantic localization and mapping device provided in an exemplary embodiment of the present disclosure.

[0029] Figure 19 This is a schematic diagram of the structure of a visual semantic localization and mapping device provided in yet another exemplary embodiment of this disclosure.

[0030] Figure 20 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0031] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0032] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0033] Application Overview

[0034] Simultaneous Localization and Mapping (SLAM) technology is a core technology in fields such as intelligent mobile platforms, autonomous driving, and consumer electronics. Its accuracy, robustness, and cost directly determine the feasibility of related products and the user experience. However, among related technologies, SLAM cannot simultaneously possess the characteristics of high accuracy, strong robustness, and low cost, making it difficult to achieve large-scale deployment in cost-sensitive scenarios with high requirements for positioning accuracy and stability. For example, while LiDAR-based SLAM systems can achieve high accuracy, the high cost and large size of LiDAR also limit the effectiveness of LiDAR-based SLAM systems. Similarly, depth camera-based SLAM, while lower in cost and smaller in size, has a limited operating range (typically no more than 10m) and is susceptible to ambient lighting interference, resulting in poor stability. Visual Simultaneous Localization and Mapping (VSLAM) and Visual Inertial Odometry (VIO) are not only lower in cost and smaller than depth camera-based SLAM systems, but they suffer from scale drift and pose shift issues. Furthermore, in scenarios where feature matching is difficult to identify (e.g., weakly textured areas) or undergoes drastic changes (e.g., rapid movement, sudden changes in lighting), the robustness of feature matching decreases, leading to tracking loss and localization interruptions.

[0035] Based on the above, this disclosure provides a method, apparatus, and electronic device for localization and mapping based on visual semantics. During the localization and mapping process, it can combine camera pose, depth information, and semantic features of target instances to achieve effective fusion of visual semantic information and three-dimensional geometric information. Based on the joint constraints of multi-source information, it can improve the accuracy and robustness of instance tracking in localization and mapping, suppress tracking loss and ID drift, and achieve long-term continuous and reliable instance tracking.

[0036] Exemplary System

[0037] Figure 1 This is a schematic diagram of a hardware system architecture for implementing positioning and mapping according to an embodiment of the present disclosure.

[0038] like Figure 1 As shown, an embodiment of this disclosure provides an electronic device 101 including at least one processor 1011 and a memory 1012.

[0039] The memory 1012 is used to store computer program instructions; the processor 1011 is configured to execute the computer program instructions stored in the memory 1012 to implement localization and mapping: based on the target frame image acquired by the camera, the camera pose and the depth map corresponding to the target frame image are determined; then, the target instance in the target frame image and the semantic features of the target instance in the target frame image are determined; based on the depth map corresponding to the target frame image and the target instance, the depth information of the target instance in the target frame image is determined; finally, based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image, the 3D point cloud map is updated. Therefore, the electronic device 101 provided in this embodiment can effectively fuse visual semantic information and 3D geometric information by combining camera pose, depth information, and the semantic features of the target instance during the localization and mapping process. It improves the accuracy and robustness of instance tracking in localization and mapping based on the joint constraints of multi-source information, suppresses tracking loss and ID drift, and achieves long-term continuous and reliable instance tracking.

[0040] The processor 1011 can be one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and a digital signal processor (DSP).

[0041] The memory 1012 may be one or more combinations of volatile or non-volatile storage media such as Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), and Flash Memory. Furthermore, the memory may also include a Solid State Drive (SSD), a Hard Disk Drive (HDD), or other non-volatile high-capacity storage devices for long-term storage of system programs, application programs, acquired data (such as images, depth maps, semantic features, point cloud data, etc.), and intermediate processing results. In some embodiments, the memory may also include removable storage media such as Secure Digital (SD) cards, MicroSD cards, Universal Serial Bus (USB) flash drives, etc., to facilitate data expansion, backup, or migration. The computer program instructions stored in the memory may include operating system, device drivers, localization and mapping applications, instance segmentation models, depth estimation models, and other necessary software modules to support electronic devices in completing the aforementioned localization and mapping tasks.

[0042] In one embodiment, the electronic device 101 further includes at least one sensor 1013. The at least one sensor 1013 is used to collect environmental information. The at least one sensor 1013 may include, for example, a camera, lidar, depth camera, or other device capable of sensing surrounding environmental data. The environmental information may include physical objects in the environment such as road signs, buildings, obstacles, pedestrians, vehicles, trees, and road surfaces. The memory 1012 is also used to store the environmental information collected by the sensor 1013; the processor 1011 is also used to process the environmental information stored in the memory 1012 when executing computer program instructions.

[0043] Electronic device 101 may be, for example, a robot, vehicle, drone, augmented reality (AR) / virtual reality (VR) device or smart terminal (such as smartphone, tablet, etc.).

[0044] The embodiments disclosed herein can be widely applied to various scenarios that require simultaneous localization and map building, such as mobile robot scenarios, autonomous driving scenarios, and augmented reality / virtual reality (AR / VR) scenarios.

[0045] For example, in a mobile robot scenario, the electronic device 101 can be a robot, which has at least one built-in processor 1011 and memory 1012. The robot may also be equipped with a sensor 1013, which can be a monocular or multi-view camera. The sensor 1013 acquires environmental images in real time and stores them in the memory 1012. The processor 1011 executes a computer program stored in the memory 1012 to process the images stored therein, using each frame of the acquired images as a target frame image, determining the camera pose in real time, and updating the 3D point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image. The robot provided in this embodiment can maintain stable instance tracking and 3D point cloud map updates under dynamic disturbances, thereby supporting the robot to achieve long-term, reliable autonomous navigation and obstacle avoidance.

[0046] For example, in an autonomous driving scenario, the electronic device 101 can be a vehicle, which has at least one built-in processor 1011 and memory 1012. Sensors 1013 can also be installed on the vehicle. Sensors 1013 acquire road images in real time and store them in memory 1012. The processor 1011 executes a computer program stored in memory to process the images stored in memory 1012, thereby using each frame of the acquired images as a target frame image, determining the camera pose in real time, and updating the 3D point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image. The real-time updated 3D point cloud map can more accurately assist the vehicle in completing tasks such as navigation and positioning, thereby improving driving safety.

[0047] For example, in an AR / VR scenario, the electronic device 101 can be an AR / VR head-mounted device. The AR / VR head-mounted device has at least one built-in processor 1011 and memory 1012. A sensor 1013 may also be installed on the AR / VR head-mounted device. The sensor 1013 captures the user's surrounding environment in real time and stores it in the memory 1012. The processor 1011 executes a computer program stored in the memory to process the images stored in the memory 1012, thereby using each frame of the captured images as a target frame image, determining the camera pose in real time, and updating the 3D point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image. The real-time updated 3D point cloud map enables more accurate positioning and navigation, enhancing the user's immersion and experience.

[0048] Exemplary methods

[0049] Figure 2This disclosure provides a flowchart of a visual semantic-based localization and mapping method. This method can be executed by any electronic device or apparatus that requires simultaneous localization and map building, such as… Figure 1 The electronic device 101 shown, or, Figure 18 The visual semantic-based localization and mapping device 1800 shown, or, Figure 20 The electronic device 11 shown. Any electronic device or apparatus includes a processor and a memory. The processor may be a CPU, GPU, or dedicated AI processor, etc. The processor executes the computer program stored in the memory to achieve localization and mapping.

[0050] like Figure 2 As shown, the method includes the following steps S210~S240.

[0051] S210: Based on the target frame image acquired by the camera, determine the camera pose and the depth map corresponding to the target frame image.

[0052] The camera in this embodiment can be a monocular camera or a multi-view camera. A multi-view camera refers to a camera system consisting of multiple cameras rigidly connected within the same base or housing. For example, a multi-view camera can include a binocular camera, a tri-view camera, or a surround-view camera. In a multi-view camera, the relative poses (i.e., extrinsic parameters, including relative rotation and relative translation) between the cameras are pre-calibrated and remain fixed. When the entire multi-view camera moves as a rigid body, the absolute poses of each camera change synchronously with the system, and the relative geometric relationships between the cameras remain constant. Based on this rigid connection characteristic, each camera can synchronously acquire images from its own fixed viewpoint, thereby providing the system with multi-view images at the same time.

[0053] In this embodiment of the disclosure, the camera may be located inside the electronic device 101 or outside the electronic device 101.

[0054] The target frame image is an image captured by the camera at any given moment. The previous frame image is the frame preceding the target frame image. Of course, the time interval between the previous frame image and the target frame image is not limited; it can be images captured at two adjacent moments, or two adjacent frames captured at a preset time interval. This disclosure does not impose any specific limitations.

[0055] In this embodiment of the disclosure, when determining the camera pose based on the target frame image, it can be determined based on the previous frame image and the target frame image. Specifically, inter-frame feature matching is performed between the target frame image and the previous frame image to obtain feature matching point pairs for the same target object; the relative pose of the target frame image captured by the camera relative to the previous frame image is calculated based on the feature matching point pairs; and the camera pose corresponding to the target frame is determined by combining the camera pose and the relative pose of the previous frame image. Of course, the motion information of the target frame and the previous frame provided by the Inertial Measurement Unit (IMU) can also be fused for joint optimization to improve the accuracy of camera pose estimation. No specific limitations are made here.

[0056] It should be noted that when the camera acquiring the target frame image is a monocular camera, the target frame image is the single-view image acquired by that monocular camera. When the camera acquiring the target frame image is a multi-camera system, the target frame image can be an image acquired by any camera in the multi-camera system from its corresponding viewpoint. Following the method described above for determining camera pose, the initial value of the inter-frame relative pose of any camera is determined. Then, based on the camera pose and relative pose of the previous frame of that camera, the camera pose of the current frame is determined. Alternatively, the camera pose of other cameras from different viewpoints can be further determined by combining the camera pose of the current frame with the camera's extrinsic parameters. The camera pose of the current frame of any camera from any viewpoint can be used as the camera pose of the target frame image. Alternatively, the camera pose of the current frame of any camera from any viewpoint can be converted to a camera pose in the multi-camera standard coordinate system. No specific limitations are made here. The multi-camera standard coordinate system can be a coordinate system defined by any viewpoint of the multi-camera system.

[0057] In this embodiment of the disclosure, the depth map is determined based on the target frame image using the following two methods: Method 1: The target frame image captured by the camera is input into a pre-trained depth prediction model (i.e., the depth predictor hereinafter), and the corresponding depth map is output. Method 2: Based on the pose transformation between adjacent frame images and the feature matching point pairs between adjacent frame images, initial depth information is obtained through triangulation, and the initial depth information is densified to determine the depth value of each pixel in the target frame image, thereby obtaining the corresponding depth map.

[0058] It should be noted that when the camera acquiring the target frame image is a monocular camera, the target frame image is the single-view image acquired by that monocular camera. When the camera acquiring the target frame image is a multi-view camera, the target frame image can be an image acquired by any of the cameras in the multi-view camera system, corresponding to the viewpoint of that camera. Following the method described above for determining the depth map, the depth maps corresponding to each viewpoint of the target frame image are determined. Alternatively, the depth maps corresponding to each viewpoint of the target frame can be fused to obtain a depth map in the multi-view camera's labeled coordinate system.

[0059] In this embodiment of the disclosure, if the target frame image is an image captured at the initial moment after the camera starts, that is, the target frame image is the first frame image captured after the camera starts, since there is no previous frame image as a reference, the camera pose cannot be determined by the above-mentioned inter-frame relative pose recursion method. In this case, the camera extrinsic parameters can be used as the camera pose of the target frame. If the target frame image is an image captured at a time other than the initial moment after the camera starts, the camera pose can be determined according to the above method.

[0060] It should be noted that in the relocation scenario after camera tracking is lost, since the camera has lost the pose association between the target frame image and the previous frame image, the relative pose cannot be calculated through the above inter-frame feature matching. In this case, the currently acquired target frame image can be matched with the key frame image in the pre-built map. Based on the matching result, the relative pose of the current target frame image relative to the key frame can be obtained. Then, based on the relative pose and the camera pose corresponding to the key frame, the camera pose of the target frame can be determined.

[0061] S220: Determine the target instance in the target frame image and the semantic features of the target instance in the target frame image.

[0062] Instance segmentation can be performed on the target frame image to obtain multiple instances of the target frame image, and the target instance in the target frame image can be one of these multiple instances.

[0063] For example, a segmentation model similar to SAM2 (Segment Anything Model 2) can be used to perform instance segmentation on the target frame image, and the object instance mask can be obtained from the output of the segmentation model to achieve accurate segmentation and boundary delineation of the object contours in the scene.

[0064] After obtaining the target instance, the instance mask corresponding to the target instance is extracted, and the semantic features of the target instance in the target frame image are extracted based on the instance mask. For example, the region of the target instance in the target frame image is determined using the instance mask, and features are extracted and aggregated in the image of that region to obtain the semantic features of the target instance.

[0065] The semantic features in this embodiment refer to the high-level feature representation of the type or semantic information corresponding to each instance.

[0066] For example, the semantic features in this embodiment can be generated based on Instance-Grounded Geometry Transformer (IGGT) features. For instance, an IGGT-like model can be used to batch process the image features of the target image frame to obtain a dense embedding vector corresponding to each pixel in the target image frame. This vector encodes the deep semantic context information of the corresponding pixel. Then, a SAM2 (Segment Anything Model 2)-like model is used to perform instance segmentation on the target frame image to obtain instance masks, which delineate the boundaries of each object. Finally, for each instance mask, the dense embedding vectors corresponding to all pixels within the instance mask are aggregated, and average pooling is used to generate the feature vector corresponding to the instance, thereby obtaining the semantic features in this embodiment.

[0067] When the dense embedding vectors corresponding to all pixels in the instance mask are aggregated and average pooling is used to generate the feature vector corresponding to the instance, the following formula (1) can be used to obtain the feature vector.

[0068] Formula (1): .

[0069] In formula (1), For the feature vector corresponding to the instance, This represents the total number of pixels within the instance mask. This represents the coordinates of all pixels covered by the instance mask. For pixels The corresponding dense embedding vector.

[0070] S230: Based on the depth map corresponding to the target frame image and the target instance, determine the depth information of the target instance in the target frame image.

[0071] Based on the instance mask corresponding to the target instance, the pixel region corresponding to the target instance in the target frame image is determined. Based on the mapping relationship between the target frame image and the depth map corresponding to the target frame image, the depth value corresponding to each pixel in the depth map is determined, thereby obtaining the depth information of the target instance in the target frame image.

[0072] S240: Update the 3D point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image.

[0073] Updating a 3D point cloud map can include: establishing a correspondence between the target instance and existing instances in the 3D point cloud map using camera pose, depth information of the target instance, and semantic features of the target instance; updating the point cloud of existing instances in the 3D point cloud map based on the correspondence, or adding the target instance as a new instance to the 3D point cloud map for updating.

[0074] In one embodiment, by fusing geometric information such as camera pose and depth information with semantic features, a matching relationship is established between the current observation (target instance) and existing instances in the 3D point cloud map to update the 3D point cloud map. For example, a forward projection association strategy can be used: combining camera pose and depth information, the target instance is back-projected from image space to 3D space to form a 3D point cloud of the target instance; subsequently, in 3D space, the semantic features and spatial position of this point cloud are matched with each instance in the 3D point cloud map, and the matching results determine whether to update each instance in the 3D point cloud map or add new instances. Alternatively, a backward projection association strategy can be used: based on camera pose, each map instance in the 3D point cloud map is projected onto the target frame image; at the 2D image level, the semantic features of the projected map instances are extracted, matched with the semantic features of the target instance, and verified using geometric constraints such as depth information, thereby establishing a matching relationship and updating the map based on the matching relationship.

[0075] The following uses the "back projection association strategy" as an example to illustrate the specific implementation process of update based on matching relationships: First, projection and feature extraction are performed. Based on the camera pose, each map instance in the 3D point cloud map is projected onto the target frame image to form a two-dimensional projection region, and the semantic features corresponding to this projection region are extracted. Next, feature matching and association determination are performed: the extracted semantic features are matched with the semantic features of the target instance in the target frame image. If the matching degree is higher than a set threshold, and the depth information of the target instance matches the expected spatial location of the map instance, then the map instance is determined to be a successful match with the target instance. Finally, map updates are performed: if there is a map instance in the 3D point cloud map that matches the target instance, the map instance in the 3D point cloud map is updated based on the depth information of the target instance in the target frame image, thus updating the 3D point cloud map; if there is no map instance in the 3D point cloud map that matches the target instance, the target instance is treated as a new instance and added to the 3D point cloud map based on the depth information of the target instance in the target frame image, thus updating the 3D point cloud map.

[0076] S210~S240 can be accessed via, for example Figure 1 The processor 1011 in the electronic device 101 shown executes the commands, or can be executed by, for example, Figure 18The processor 1801 in the visual semantic-based localization and mapping device shown executes this function, or it can be performed by, for example... Figure 20 The processor 111 in the electronic device 11 shown executes the commands.

[0077] The visual semantic-based localization and mapping method provided in this disclosure can determine the camera pose and the corresponding depth map of the target frame image based on the target frame image acquired by the camera; then, determine the target instance in the target frame image and the semantic features of the target instance in the target frame image; based on the depth map corresponding to the target frame image and the target instance, determine the depth information of the target instance in the target frame image; finally, update the 3D point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image. Therefore, this disclosure can effectively fuse visual semantic information and 3D geometric information by combining camera pose, depth information, and the semantic features of the target instance during the localization and mapping process. It improves the accuracy and robustness of instance tracking in localization and mapping based on the joint constraints of multi-source information, suppresses tracking loss and ID drift, and achieves long-term continuous and reliable instance tracking.

[0078] In some embodiments, such as Figure 3 As shown, the localization and mapping method based on visual semantics can be divided into a front-end execution part and a back-end execution part. The front-end execution part corresponds to S210 above, and the back-end execution part corresponds to S220~S240 above.

[0079] It should be noted that the "front-end execution part" and "back-end execution part" in this embodiment are logical divisions based on algorithm flow and data processing timing, rather than physical limitations on hardware. In terms of hardware implementation, the front-end execution part and the back-end execution part can run on the same processor (such as a CPU, GPU, NPU, or dedicated SLAM chip) or they can be distributed across different processors. For example, the front-end execution part (corresponding to S210) involves real-time acquisition and pose tracking of camera images, which has high real-time requirements and can be accelerated by a dedicated image processor or GPU; the back-end execution part (corresponding to S220~S240) involves instance segmentation, semantic feature extraction, and map updating, which involves a large amount of computation but has relatively low real-time requirements and can be executed by a CPU or a separate computing unit. This embodiment does not limit this aspect.

[0080] In one exemplary hardware deployment, the front-end execution unit runs on a processor with high real-time requirements (such as a real-time CPU core or GPU) to process images acquired by the camera in real time and output camera pose and depth maps. The back-end execution unit runs on a processor with high computing power (such as a high-performance CPU, automotive-grade computing unit, or cloud server) to perform computationally intensive tasks such as instance segmentation, semantic feature extraction, and 3D point cloud map updates. The camera pose and depth maps output by the front-end are passed to the back-end via shared memory or a communication interface.

[0081] like Figure 3 As shown, in the front-end execution section, depth maps and camera poses of multiple cameras corresponding to target frame images from multiple viewpoints can be determined (the specific determination method can be found in the corresponding description in the subsequent embodiments, which will not be described in detail here). In the back-end execution section, the instance mask and depth information of the target instance in the target frame image can be obtained; the target frame image is input into the semantic feature predictor, and the semantic features of the target instance are obtained through the semantic feature predictor; finally, based on the depth information of the target instance, the camera poses of multiple cameras, and the semantic features of the target instance, the 3D point cloud map is updated through two-stream instance matching of semantic and geometric constraints (corresponding to S240).

[0082] In this embodiment of the disclosure, the semantic feature predictor may be a model trained by deep learning, used to extract IGGT features corresponding to the target instance from the target frame image.

[0083] In this embodiment, the dual-stream instance matching based on semantic and geometric constraints (corresponding to S240) can effectively fuse visual semantic information and three-dimensional geometric information, thereby enhancing the stability of instance association between different frames. Specifically, semantic features provide constraints on instance category and appearance consistency, while geometric information (including camera pose and depth data) ensures the accuracy of instance spatial location. The combination of the two can effectively filter out erroneous matches caused by changes in illumination, differences in viewpoint, or local occlusion, significantly reducing the probability of ID drift during instance tracking.

[0084] The following section will first introduce the two-stream instance matching of semantic and geometric constraints (corresponding to S240).

[0085] In some embodiments, such as Figure 4 As shown, the execution process of S240 may include the following S410~S430.

[0086] S410: Based on the camera pose and the instance mask of the target instance in the target frame image, match the target instance with a first instance of the 3D point cloud map to determine a first matching result; and / or, based on the semantic features of the target instance in the target frame image, match the target instance with a first instance of the 3D point cloud map to determine a second matching result.

[0087] The first instance refers to any instance among the existing map instances stored in the 3D point cloud map. It should be noted that the 3D point cloud map stores one or more first instances. During the matching process, for each target instance, it is matched with one or more first instances to obtain the corresponding matching result (i.e., the first matching result and / or the second matching result). This matching result indicates whether a first instance matches the target instance or whether no first instance matches the target instance. In multi-camera scenarios, the aforementioned matching process is performed independently for each target frame image (or each viewpoint image). Therefore, any target instance may successfully match a first instance or fail to match any first instance; conversely, any first instance may be associated and matched with target instances in one or more frames. In the specific matching process, cross-viewpoint fusion can be performed on the matching results from different frames or different viewpoints to improve the robustness of the association.

[0088] The first matching result refers to the matching result obtained based on geometric information (camera pose and instance mask), used to indicate whether there is a first instance that matches the target instance geometrically and the matching instance object; the second matching result refers to the matching result obtained based on semantic information (semantic features), used to indicate whether there is a first instance that matches the target instance semantically and the matching instance object. The matching result includes successful matching or failed matching.

[0089] In one possible embodiment, the matching between the target instance and the first instance is performed through a geometric consistency matching path and / or a semantic similarity matching path to obtain a matching result. Specifically, the geometric consistency matching path involves projecting the first instance onto the target frame image based on the camera pose, and performing spatial matching analysis between the projection result and the instance mask of the target instance to obtain a first matching result. The semantic similarity matching path involves calculating the similarity between the semantic features of the target instance and the semantic features of the first instance, and determining a second matching result based on the similarity.

[0090] S420: Determine the target matching result based on the first matching result and / or the second matching result.

[0091] In this embodiment of the disclosure, the matching result can be determined based on geometric consistency matching or semantic similarity matching respectively, or the two matching results can be fused to obtain the final matching result. This embodiment of the disclosure does not limit this.

[0092] In some optional embodiments, when combining the first matching result and the second matching result to determine the target matching result, the first matching result can be determined first. If the first matching result is a successful match, it can be directly used as the target matching result; if the first matching result is a failed match, then the process of determining the second matching result is executed (e.g., Figure 6 (The dashed arrows point to the corresponding processes). That is, geometric tracking is performed first, and semantic tracking is only performed if geometric tracking fails. Specifically, geometric tracking relies on the spatial projection relationship between the camera pose and the instance mask, has low computational complexity, and is fast in matching, quickly filtering out candidate instances with spatial overlap. Semantic tracking, on the other hand, uses deep semantic feature similarity matching, enabling effective instance identification and association even when geometric information is insufficient (such as occlusion or drastic changes in viewpoint). This dual-stream matching strategy of "geometry first, semantic supplementation" significantly improves the success rate and stability of instance matching while ensuring matching efficiency.

[0093] In some embodiments, the target matching result can be determined solely based on the first matching result, performing only geometric consistency matching and setting the first matching result as the target matching result. This matching method relies solely on the spatial projection relationship between camera pose and instance mask for instance association, without involving the extraction and comparison of semantic features. This method has low computational cost and high real-time performance, making it suitable for scenarios with sufficient geometric information, minimal occlusion, and high real-time requirements.

[0094] In some embodiments, the target matching result can be determined solely based on the second matching result, performing only semantic similarity matching and setting the second matching result as the target matching result. This matching method associates instances based on the similarity between semantic feature vectors, and is suitable for scenarios with obvious semantic features but weak geometric constraints.

[0095] S430: Update the 3D point cloud map based on the target matching results, the camera pose, the depth information of the target instance in the target frame image, and the instance mask of the target instance in the target frame image.

[0096] If the target matching result is successful, the target instance is fused and updated with the first matched instance in the 3D point cloud map. That is, based on the camera pose and depth information, the target instance is back-projected into 3D space to obtain the 3D point cloud of the target instance; this 3D point cloud is then fused with the existing 3D point cloud of the first instance (such as point cloud stitching, geometric averaging, or weighted fusion) to optimize the geometric structure and semantic features of the instance.

[0097] If the target matching result fails, the target instance is added as a new instance to the 3D point cloud map. That is, based on the camera pose and depth information, the target instance is back-projected into 3D space to generate a new 3D instance point cloud, and a new instance identifier (ID) is assigned to it. At the same time, its semantic features are associated to form the semantic and geometric information of the new instance, and the new instance is fused into the 3D point cloud map, thereby completing the update of the 3D point cloud map.

[0098] Through steps S410~S430, geometric and / or semantic matching is used to match the current observation with the map instance. The 3D point cloud map is updated based on the matching results to achieve geometric and semantic complementarity and improve matching accuracy. Furthermore, geometric tracking or geometric matching is used to ensure matching efficiency, and then semantic matching is used to improve the matching success rate, ultimately achieving dynamic updating of the 3D point cloud map.

[0099] In some embodiments, such as Figure 5 As shown, based on the camera pose and the instance mask of the target instance in the target frame image, the target instance is matched with the first instance of the 3D point cloud map to determine the first matching result, which may include the following steps S510~S530.

[0100] S510: Based on the camera pose, project each first instance in the 3D point cloud map onto the target frame image under the corresponding view of the camera, and determine the number of first projection points of each first instance falling into the instance mask of the target instance in the target frame image.

[0101] For example, such as Figure 6 As shown, a 3D-2D projection of a 3D point cloud map can be performed based on the camera pose and the target frame image. That is, the 3D point cloud of each first instance in the 3D point cloud map is transformed from the world coordinate system to the camera coordinate system. Then, according to the pre-calibrated intrinsic parameters of the camera (focal length, principal point, etc.) and the preset projection method (such as perspective projection, orthographic projection, or affine projection), the 3D points of the first instance in the camera coordinate system are projected onto the target frame image to obtain the image projection result of each first instance in the corresponding viewpoint of the camera. For each first instance, the number of projection points of the first instance that fall into the instance mask corresponding to the target instance is counted to obtain the number of first projection points corresponding to the first instance.

[0102] S520: In response to the fact that the number of first projection points of each first instance is less than the first preset threshold, the first matching result is determined to be a matching failure.

[0103] For example, such as Figure 6 As shown, the projection of each first instance onto the target frame image (i.e. Figure 6 Geometric tracking is performed on the projected point cloud. Specifically, the number of first projection points of each first instance in the 3D point cloud map is compared with a first preset threshold. If the number of first projection points of each first instance in the 3D point cloud map is less than the first preset threshold, it indicates that there is not enough overlapping projection area between the target instance and any instance in the 3D point cloud map. The matching degree between the target instance and any instance in the 3D point cloud map is low. Therefore, it can be determined that the target instance fails to match any of the first instances in the 3D point cloud map, i.e., geometric tracking matching fails.

[0104] In some embodiments, in the event of geometric tracking matching failure, such as Figure 6 As shown by the dashed arrow pointing from geometric tracking to semantic tracking, the target instance can be matched with each first instance in the 3D point cloud map through semantic tracking. The matching process can be referred to the relevant description in the subsequent corresponding embodiments (corresponding to S710~S740), which will not be elaborated here.

[0105] The first preset threshold can be set by those skilled in the art according to the actual situation, and the embodiments disclosed herein do not limit this.

[0106] S530: In response to the existence of one or more first instances where the number of first projection points is greater than or equal to a first preset threshold, determine the number of first projection points corresponding to each first instance, determine the first instance with the largest number of first projection points as the matching first instance that matches the target instance, and determine the first matching result as a successful match.

[0107] If the number of first projection points of one or more first instances is greater than or equal to a first preset threshold, it indicates that there is a significant overlapping projection area between the one or more first instances and the target instance, and the matching between the one or more first instances and the target instance is high. In this case, the maximum value among the number of first projection points of one or more first instances can be selected, and the first instance corresponding to the maximum value can be determined as the matching first instance that matches the target instance. Then, the final matching result is determined according to any of the following methods: 1. Based on the matching first instance, the first matching result is determined to be a successful match; 2. Further verification is performed using semantic features. If the semantic verification passes, the match is determined to be successful; otherwise, the match is determined to be unsuccessful; 3. Multiple first instances with the top N (N is a positive integer and N is greater than or equal to 2) in terms of the number of projection points are used as a candidate set, and semantic verification is performed on each of them. The candidate instance with the highest semantic similarity is selected as the matching first instance. In subsequent semantic similarity determination, if the semantic similarity between the matching first instance and the target instance is lower than the corresponding semantic threshold, the match is determined to be unsuccessful.

[0108] S510~S530 can be accessed via, for example Figure 1 The processor 1011 in the electronic device 101 shown executes the commands, or can be executed by, for example, Figure 18 The processor 1801 in the visual semantic-based localization and mapping device shown executes this function, or it can be performed by, for example... Figure 20 The processor 111 in the electronic device 11 shown executes the commands.

[0109] In some embodiments, such as Figure 7 As shown, based on the semantic features corresponding to the target instance in the target frame image, the target instance is matched with each first instance of the 3D point cloud map to determine the second matching result, including the following S710~S740.

[0110] S710: Calculate the first similarity of the semantic features corresponding to each first instance and the target instance in the target frame image, and obtain the first similarity corresponding to each first instance.

[0111] For example, such as Figure 6 As shown, based on the instance mask corresponding to the target instance, a target frame image with an instance mask is obtained. Based on the target frame image with an instance mask and its semantic features, mask-level extraction is performed on the semantic features of the target frame image. That is, the semantic features corresponding to the instance mask of each first instance in the target frame image are extracted to obtain a semantic feature mask. Based on the semantic feature mask, the following steps are performed: Figure 6 The semantic tracking shown is to calculate the first similarity between the semantic feature mask corresponding to each first instance and the semantic features of the target instance, and obtain the first similarity corresponding to each first instance.

[0112] In some examples, the semantic feature mask corresponding to each first instance can be matched with the semantic features corresponding to the target instance by calculating cosine similarity, Euclidean distance, etc., to obtain the first similarity between each first instance and the target instance.

[0113] S720: Determine the maximum similarity among the first similarities corresponding to each first instance.

[0114] S730: In response to the maximum similarity being greater than or equal to the second preset threshold, the second matching result is determined to be a successful match.

[0115] If the maximum similarity is greater than or equal to the second preset threshold, it indicates that the first instance corresponding to the maximum similarity and the target instance have a high degree of consistency in semantic features, and the confidence that they belong to the same object is high. Therefore, the second matching result is determined to be a successful match.

[0116] S740: In response to the maximum similarity being less than the second preset threshold, the second matching result is determined to be a matching failure.

[0117] If the maximum similarity is less than the second preset threshold, it indicates that the first instance and the target instance are significantly different in semantic features and do not have sufficient semantic matching degree. Therefore, the second matching result is determined to be a matching failure.

[0118] The maximum value is selected from the first similarity values ​​corresponding to each first instance. Based on the maximum similarity, the matching result is determined, including: 1. If the maximum similarity is ≥ the second preset threshold, the matching is determined to be successful; otherwise, the matching is determined to be unsuccessful; 2. If the maximum similarity is ≥ the second preset threshold, the difference between the maximum similarity and the similarity value that is second only to the first similarity is further compared. If the difference is greater than or equal to the preset difference threshold (i.e., the maximum similarity is significantly higher than other similarities), the matching is determined to be successful; if the difference is less than the preset difference threshold (i.e., the similarity of multiple instances is very close), the matching is temporarily determined to be unsuccessful.

[0119] The embodiments disclosed herein do not limit the semantic matching success determination strategy, and those skilled in the art can flexibly set it according to actual application scenarios.

[0120] In some embodiments, the first matching result can be determined as the target matching result in response to the first matching result being a successful match; and the second matching result can be determined as the target matching result in response to the first matching result being a failed match.

[0121] In some embodiments, the processes of determining a first matching result and determining a second matching result can be executed simultaneously, and then the target matching result can be determined based on the first matching result and the second matching result.

[0122] For example, if both the first and second matching results are successful, the target matching result is determined to be a successful match. If either the first or second matching result is a failed match, preset weights are configured for the maximum value of the number of first projection points in each first instance and the maximum similarity obtained during the calculation of the second matching result. A weighted sum is calculated based on the maximum preset weight and the maximum similarity to obtain a comprehensive matching score. If the comprehensive matching score is greater than or equal to a first preset comprehensive threshold, the target matching result is determined to be a successful match. If the comprehensive matching score is less than the first preset comprehensive threshold, the target matching result is determined to be a failed match. The first preset comprehensive threshold can be set by those skilled in the art according to actual conditions, and this embodiment does not limit this setting.

[0123] In this embodiment, dual verification of geometric spatial constraints and semantic feature consistency can be achieved, further improving the accuracy and reliability of instance matching. When there are certain differences between the results of geometric matching and semantic matching, a weighted fusion method is used to combine the advantages of both. This avoids the limitations of a single matching method and makes full use of the complementarity of multi-source information, making the instance matching results more accurate. This comprehensive matching strategy is particularly suitable for localization and mapping tasks in complex dynamic environments, such as scenarios with frequent occlusion, sudden changes in illumination, or rapid switching of viewpoints. It can effectively cope with various interference factors and ensure the stability and accuracy of 3D point cloud map updates.

[0124] For example, if both the first and second matching results are successful, the target matching result is determined to be a successful match; if either the first or second matching result fails, the target matching result is determined to be a failed match. Based on this, rigorous verification of instance matching results can be achieved, ensuring that only instances that simultaneously satisfy geometric consistency and semantic feature consistency are considered successful matches, thereby reducing the possibility of erroneous matching. This "double success" judgment rule can effectively filter out erroneous matches caused by accidental spatial overlap or semantic similarity, further improving the accuracy and reliability of 3D point cloud map updates, and providing a more solid foundation for subsequent localization and mapping tasks. In practical application scenarios, such as navigation in complex indoor environments or equipment monitoring in industrial scenarios, this strict matching rule can significantly reduce map drift problems caused by erroneous instance associations, ensuring the stability and accuracy of the system during long-term operation.

[0125] For example, if the second matching result is a failed match, the target matching result is determined to be a failed match; if the second matching result is a successful match, the target matching result is further determined based on the first matching result: if the first matching result is a successful match, the target matching result is determined to be a successful match; if the first matching result is a failed match, the target matching result is determined to be a failed match. Based on this, priority verification of semantic feature consistency can be achieved, ensuring the similarity of matched instance attributes. Since semantic features can capture the deep category information and appearance features of target instances, even when geometric projection fails due to factors such as occlusion, drastic changes in viewpoint, or motion blur, the correct matching instance can still be selected through semantic similarity judgment. This "semantic-driven, geometrically assisted" strategy fully leverages the robustness of semantic features and further verifies the spatial rationality of the matching with geometric information, thereby effectively improving the accuracy and stability of instance matching in complex dynamic scenes and increasing the accuracy of real-time updates of 3D point cloud maps.

[0126] S710~S740 can be accessed via, for example Figure 1 The processor 1011 in the electronic device 101 shown executes the commands, or can be executed by, for example, Figure 18 The processor 1801 in the visual semantic-based localization and mapping device shown executes this. Alternatively, it can be performed via, as in... Figure 20 The processor 111 in the electronic device 11 shown executes the commands.

[0127] In some embodiments, if the target matching result is a successful match, such as Figure 8 As shown, step S430 further includes S810~S820.

[0128] S810: Based on the camera pose, the depth information of the target instance in the target frame image, and the instance mask of the target instance in the target frame image, the target instance is back-projected onto the map coordinate system corresponding to the 3D point cloud map to obtain the matching instance point cloud of the target instance in the map coordinate system.

[0129] For example, such as Figure 6 As shown, based on the camera pose, the instance mask of the target instance in the target frame image can be back-projected to the map coordinate system (i.e., world coordinate system) corresponding to the 3D point cloud map. According to the depth information corresponding to the instance mask, the 3D coordinates of the pixels of the target instance are calculated to obtain the 3D point cloud set of the target instance in the map coordinate system, i.e., the matching instance point cloud.

[0130] For example, for each pixel within the instance mask of the target instance, the pixel coordinates and corresponding depth value in the target frame image can be obtained first. Then, combined with pre-calibrated camera intrinsic parameters, the pixel coordinates are converted into 3D coordinates in the camera coordinate system. Using the rotation matrix R and translation vector T corresponding to the camera pose, the 3D coordinates in the camera coordinate system are converted into coordinates in the map coordinate system (world coordinate system). Finally, all the converted 3D coordinate points are aggregated to obtain the matching instance point cloud of the target instance in the map coordinate system.

[0131] S820: Merge the matching instance point cloud with the first instance point cloud corresponding to the first matching instance in the 3D point cloud map to obtain the updated instance point cloud corresponding to the first matching instance; wherein, the first matching instance is the first instance in the 3D point cloud map that matches the target instance.

[0132] For example, the first instance in the 3D point cloud map that matches the target instance is determined as the first matching instance, and the instance mask of the target instance is associated with the ID of the first matching instance to establish a correspondence between the two (correspondence). Figure 6 (Selection of mask ID). After associating the instance mask of the target instance with the ID of the first matching instance, the point cloud of the matching instance corresponding to the instance mask of the target instance can be concatenated or weighted and fused with the point cloud of the first matching instance to obtain the updated instance point cloud corresponding to the first matching instance.

[0133] The method of stitching or weighted fusion of the point cloud of the matching instance with the point cloud of the first instance corresponding to the first matching instance can be set by those skilled in the art according to the actual situation. For example, duplicate and redundant points can be removed, and valid point clouds with consistent spatial position and geometric features can be retained. The fused point cloud can be filtered and optimized to obtain the updated instance point cloud corresponding to the first matching instance. This disclosure does not limit this method.

[0134] S810~S820 can be used as follows Figure 1 The processor 1011 in the electronic device 101 shown executes the commands, or can be executed by, for example, Figure 18 The processor 1801 in the visual semantic-based localization and mapping device shown executes this. Alternatively, it can be performed via, as in... Figure 20 The processor 111 in the electronic device 11 shown executes the commands.

[0135] In some embodiments, if the target matching result is a match failure, such as Figure 9 As shown, step S430 further includes steps S910 to S920.

[0136] S910: Based on the camera pose, the depth information of the target instance in the target frame image, and the instance mask of the target instance in the target frame image, the target instance is back-projected onto the map coordinate system corresponding to the 3D point cloud map to obtain the matching instance point cloud of the target instance in the map coordinate system.

[0137] The execution process of S910 is the same as that of S810. Please refer to the corresponding description of the relevant embodiments in S810, which will not be repeated here.

[0138] S920: Add the matching instance point cloud to the 3D point cloud map.

[0139] If the target matching result fails, it means that the target instance is a new unmarked instance in the 3D point cloud map. In this case, instance identification information can be created for the target instance, including assigning an instance ID, semantic features associated with the target instance, and metadata such as the collection timestamp. Then, the matching instance point cloud is used as the initial point cloud data of the target instance, bound to the instance identification information corresponding to the target instance, and added to the instance list of the 3D point cloud map. Finally, the matching instance point cloud is merged into the 3D point cloud map to obtain the updated 3D point cloud map.

[0140] S910~S920 can be accessed via, for example Figure 1 The processor 1011 in the electronic device 101 shown executes the commands, or can be executed by, for example, Figure 18 The processor 1801 in the visual semantic-based localization and mapping device shown executes this. Alternatively, it can be performed via, as in... Figure 20 The processor 111 in the electronic device 11 shown executes the commands.

[0141] In dynamic environments, due to factors such as changes in lighting, target motion, background interference, and sensor noise, the geometric and appearance features of the same target instance may differ significantly at different times. This makes it difficult for the system to correctly determine whether observation data in different frames belong to the same instance, resulting in the same instance being fragmented into multiple isolated segments with different identifiers (IDs) in the temporal dimension, forming semantic information breaks and compromising the temporal consistency of scene understanding and the integrity of the map. To prevent semantic information breaks, in some embodiments, such as... Figure 10 As shown, determining the semantic features of the target instance in the target frame image may include the following steps S1010~S1020.

[0142] S1010: Based on a sliding window with a preset sliding step size, determine the identifier and semantic features of each second instance in the target frame image within the current sliding window.

[0143] In this embodiment of the disclosure, the instances detected in the target frame images within the current sliding window are denoted as second instances, to distinguish them from the first instances in the 3D point cloud map. The second instances include all instances appearing in all target frame images within the current window, and each second instance corresponds to an instance identifier and a set of semantic features.

[0144] The preset sliding step size can be configured according to the dynamic changes of the actual scene. For example, in a dynamic scene with dense pedestrians, the preset sliding step size is 5 frames to ensure effective capture of rapidly changing instance features; in a static scene, the preset sliding step size can be appropriately increased to 10 frames to reduce the consumption of computing resources.

[0145] S1020: Based on the identifiers and semantic features of each third instance in the anchor frame image of the previous sliding window of the current sliding window, update the identifiers and semantic features of each second instance in the target frame image.

[0146] The third instance refers to the instance detected in the anchor frame image within the previous sliding window. The anchor frame is one or more frames selected from the previous sliding window, serving as the benchmark for cross-window semantic feature alignment. Each third instance corresponds to an instance identifier and a set of semantic features, used for association and matching with the second instance in the current sliding window, to achieve cross-window transfer and updating of instance identifiers and semantic features.

[0147] You can select at least one frame as the anchor frame from multiple frames within the previous sliding window, based on a preset number of anchor frames. The preset number of anchor frames is less than the number of frames in the previous sliding window.

[0148] For example, a time uniform sampling strategy can be adopted, in which one frame is selected as the anchor frame image at every preset interval in the m1 frame images contained in the previous sliding window (m1 is the number of window frames, m1 is a positive integer), to ensure that the anchor frame images are uniformly distributed in the time dimension; or the highest quality m2 frame is selected as the anchor frame image based on the image quality score (such as sharpness, number of feature points) (m2 is the preset number of anchor frames, m2 is a positive integer) to improve the reliability of feature matching.

[0149] After determining the anchor frame image, a second similarity is calculated between the semantic features of the third instance in the anchor frame image and the semantic features of the second instances in each target frame image within the current sliding window. For example, the cosine similarity between the semantic features of the third instance in the anchor frame image and the semantic features of the second instances within the current sliding window can be calculated to obtain the second similarity between the third instance and each of the second instances.

[0150] After obtaining the second similarity between the semantic features of the third instance and the semantic features of the second instances in each target frame image within the current sliding window, each second similarity is compared with a third preset threshold. In response to a second similarity greater than the third preset threshold, the identifier and semantic features of the second instance corresponding to the second similarity are updated based on the identifier and semantic features of the third instance.

[0151] For example, the identifier of the third instance is directly assigned to the second instance to achieve instance identifier inheritance across sliding windows; and the semantic features of the third instance are directly assigned to the second instance to achieve semantic feature association and updating across sliding windows.

[0152] In this embodiment, when a target instance temporarily loses its features due to brief occlusion, rapid movement, or sudden changes in illumination within a continuous sliding window, the historical semantic features and instance identifiers stored in the anchor frame can serve as "memory anchors" to quickly recover the ID and semantics of the lost instance. This avoids the generation of new instance identifiers due to temporary feature loss, thereby maintaining the continuity of instance tracking throughout the scene. Furthermore, through feature mapping of the anchor frame, it is unnecessary to repeatedly detect the semantic features of all instances in each sliding window. Instead, the semantic feature information of existing instances in the anchor frame can be reused, significantly reducing the consumption of computational resources and effectively ensuring the stability, efficiency, and accuracy of localization and mapping tasks in complex dynamic environments.

[0153] In some embodiments, if multiple third instances have a second similarity greater than a third preset threshold with the same second instance, the third instance with the highest second similarity is selected as the benchmark, or multi-source fusion is performed on the features of all third instances that meet the conditions.

[0154] In this embodiment, the 3D point cloud map is updated based on multimodal information composed of geometric and semantic features. This improves the accuracy of matching results and adapts to scenarios with varying degrees of complexity. The combination of a sliding window and anchor frame mechanism maintains the temporal continuity of instance tracking, avoids semantic information breaks, and provides a stable foundation for instance association in localization and mapping tasks. Simultaneously, point cloud fusion or new instance addition is performed for different matching success and failure scenarios. This updates the point cloud data of existing instances in the 3D point cloud map and allows for timely addition of new instances, ensuring the integrity and real-time performance of the map. Ultimately, this provides efficient and robust backend support for localization and mapping systems in complex dynamic environments, significantly improving the overall system performance.

[0155] In the process of determining the first matching result, 3D-2D projection is performed, and in the process of updating the 3D point cloud map, 2D-3D back projection is performed. If the camera pose is inaccurate during projection and back projection, it will directly lead to spatial alignment deviation between the 3D point cloud and the 2D image features, thereby causing misjudgment of instance matching or causing the position of the 3D point cloud obtained by back projection to shift, reducing the accuracy of map update. At the same time, the depth information of the target frame image will also affect the quality and accuracy of the matching instance point cloud. For example, if the depth information has noise, is missing, or has measurement deviation (such as abnormal depth values ​​caused by reflective surfaces such as glass and mirrors), it will cause the matching instance point cloud obtained by back projection to be sparse, have positional shifts, or have invalid points, thereby interfering with point cloud fusion or causing the initial point cloud data of the new instance to be inaccurate. Based on the above problems, the embodiments of this disclosure further optimize the front-end execution part to obtain more accurate camera pose and depth information of the target frame image. The optimization of the front-end execution part of the embodiments of this disclosure is described below.

[0156] S1010~S1020 can be accessed via, for example Figure 1 The processor 1011 in the electronic device 101 shown executes the commands, or can be executed by, for example, Figure 18 The processor 1801 in the visual semantic-based localization and mapping device shown can perform this operation, or it can be performed by, for example... Figure 20 The processor 111 in the electronic device 11 shown executes the commands.

[0157] like Figure 11 As shown, the execution process of S210 may include the following S1110~S1150.

[0158] S1110: Determine the depth map of the target frame image based on the target frame image and the intrinsic and extrinsic parameters of the camera that acquired the target frame image.

[0159] For example, such as Figure 12 As shown, a depth predictor can be pre-set, and the target frame image and the camera's intrinsic and extrinsic parameters of the target frame image are input into the depth predictor to obtain the depth map of the target frame image output by the depth predictor. The depth predictor can be a model trained based on deep learning.

[0160] In some embodiments, the depth predictor and the semantic feature predictor can be the same model. In this case, the depth map of the target frame image and the semantic features of the target instance (such as IGGT features) can be output simultaneously, thereby completing two key tasks in a single inference process, effectively reducing the consumption of computing resources and inference time. For example, a joint prediction model built using a multi-task learning framework has an encoder that shares a feature extraction module such as a convolutional neural network (CNN) or a Transformer to perform high-level feature encoding on the input target frame image; the decoder generates the depth map and semantic features respectively. This joint prediction approach not only reduces the complexity of model deployment but also improves prediction accuracy by utilizing complementary information between tasks. Depth information can assist in the spatial localization of semantic features, while semantic features can provide category-level prior knowledge for depth estimation, thereby enhancing the robustness of both tasks. In addition, the inference process of the joint model is simpler, which helps reduce the latency of front-end data processing and provides more timely and accurate input data for back-end instance matching and map updating, further ensuring the real-time performance and reliability of the entire localization and mapping system.

[0161] S1120: Perform feature extraction and feature association on the target frame image and the previous frame image to obtain multiple associated point pairs in adjacent frames.

[0162] Based on the feature extraction model, image feature points of the target frame image and the image feature points of the previous frame image can be extracted.

[0163] The basic feature extraction model can be a deep learning-based feature extraction model, such as SuperPoint, D2-Net, or a feature extraction network based on the Transformer architecture. These models can automatically learn high-level semantic and geometric features in images through end-to-end training, and have stronger robustness and adaptability, making them suitable for feature association tasks in complex and dynamic scenes.

[0164] For example, the SuperPoint model can be used to extract image feature points. Image feature points can include keypoint coordinates and their corresponding feature descriptors. The SuperPoint model includes a convolutional neural network (CNN) encoder, a keypoint head, and a descriptor head. When processing an input image, SuperPoint first performs multi-scale feature extraction on the input image through the CNN encoder to obtain a multi-scale feature map; the keypoint head and the descriptor head share the feature map output by the CNN encoder. The keypoint head, based on a heatmap prediction mechanism, predicts the probability that each spatial location is a keypoint on the feature map, and filters out keypoints according to a preset confidence threshold, outputting the keypoint coordinates and their confidence scores. The descriptor head uses a contrastive learning strategy to generate a corresponding high-dimensional descriptor vector for each spatial location in the feature map. Based on the spatial location of the keypoint in the feature map, it indexes the corresponding descriptor vector for each keypoint in the generated high-dimensional descriptor vector as the feature descriptor for that keypoint. Through the heatmap prediction mechanism, stable feature points with high response values ​​(such as corner points and edge intersections) in the image can be accurately located, effectively filtering invalid points in noise or low-contrast areas. Through the contrastive learning strategy, the descriptors of the same feature point under different lighting and viewing angle changes have high similarity, while the descriptors of different feature points maintain significant distinguishability, thereby ensuring the repeatability and reliability of feature points in cross-frame matching, enhancing the robustness of feature points in complex scenes, and thus outputting high-quality and repetitive feature points under conditions of lighting changes and viewing angle differences. Through the above-mentioned shared feature map and index mapping mechanism, the consistency of key points and their descriptors in spatial location is guaranteed, thereby ensuring the accuracy and efficiency of feature extraction.

[0165] After obtaining the image feature points of the target frame image and the image feature points of the previous frame image, feature association processing can be performed on the image feature points of the target frame image and the image feature points of the previous frame image based on the feature matching basic model to obtain multiple associated point pairs.

[0166] The feature matching base model can be a deep learning-based feature matching model, such as the feature matching network of LightGlue. LightGlue can stably output high-quality associated point pairs when facing rapid movement, sudden changes in lighting, or partial occlusion in dynamic scenes, providing a geometric constraint basis for the accurate estimation of camera pose.

[0167] When performing feature association, such as Figure 13 The diagram shows the process of SuperPoint feature extraction and LightGlue feature matching. Wherein, F... k F represents the multi-view camera used to acquire target frame images. k-1 F represents the multi-view camera that captured the previous frame of the target frame image.k The 0s, 1s, 2s, and 3s surrounding F represent the cameras in a multi-target camera system that captured target frame images from different viewpoints; F k-1 The 0, 1, 2, and 3 surrounding the symbol represent the previous frame image captured by a multi-target camera from different perspectives. The triangular labels represent the image feature points extracted by the SuperPoint model (which can also correspond to map points in a 3D point cloud map).

[0168] like Figure 13 As shown, during feature association processing, Lightglue can be used to track image feature points from the previous frame, associating image feature points in target frame images taken from different camera viewpoints that are identical to those in the previous frame. For example... Figure 13 The image feature point at point ① shown in the figure is F. k Camera No. 2 and F k-1 The two image feature points corresponding to the same three-dimensional feature point in space captured by camera 0 (these two image feature points overlap, therefore...) Figure 13 (Displayed as a feature point), and associate the two image feature points; alternatively, Lightglue can be used to track image feature points of the previous frame, associating image feature points in the target frame captured by the same camera from the same viewpoint with those in the previous frame, for example... Figure 13 The image feature point at point ② shown in the figure is F. k Camera No. 3 and F k-1 The two image feature points corresponding to the same three-dimensional feature point in space captured by camera number 3 (these two image feature points overlap, therefore...) Figure 13 The two image feature points (displayed as a single feature point) are associated. Two image feature points associated in the target frame image and the previous frame image constitute an associated point pair. Through the above cross-viewpoint and same-viewpoint feature association methods, the spatial complementarity and temporal continuity of multi-view cameras can be fully utilized to obtain a richer and more robust set of associated point pairs. These associated point pairs not only contain the spatial correspondence of the same instance under different viewpoints, but also retain the temporal evolution information of the instance under the same viewpoint, effectively improving the accuracy and stability of camera pose estimation.

[0169] S1130: Based on multiple associated point pairs and the camera's intrinsic and extrinsic parameters, determine the 3D image feature points corresponding to the target frame image and the 3D image feature points corresponding to the previous frame image of the target frame image.

[0170] For example, for each pair of associated points, the pixel coordinates of the camera corresponding to the target frame image and the pixel coordinates of the camera corresponding to the previous frame image are extracted. These are then combined with a pre-calibrated camera intrinsic parameter matrix K and the preliminary estimated camera pose transformations (rotation matrix R and translation vector T) from adjacent frames obtained through feature association to construct a triangulated mathematical model. This triangulated mathematical model is then solved using a linear triangulation method (such as the Direct Linear Transformation (DLT) algorithm) to obtain the coordinates of the 3D image feature points corresponding to the pair of associated points in the world coordinate system.

[0171] S1140: Based on multiple associated point pairs, the three-dimensional image feature points corresponding to the target frame image, and the three-dimensional image feature points corresponding to the previous frame image of the target frame image, determine the first relative pose of the camera capturing adjacent frame images.

[0172] For example, such as Figure 12 As shown, multiple associated point pairs, the 3D image feature points corresponding to the target frame image, and the 3D image feature points corresponding to the previous frame image can be input into the Perspective-n-Point (PnP) odometry. Using the PnP algorithm, the 3D image feature points corresponding to the target frame image are used as reference points in the world coordinate system, and the pixel coordinates of the associated point pairs in the target frame image are used as the image plane observation points. Combined with camera intrinsic parameters, the initial relative pose of the camera corresponding to the target frame image relative to the camera corresponding to the previous frame image is solved, thus obtaining the first relative pose of each camera capturing adjacent frame images.

[0173] S1150: Determine the camera pose based on the first relative pose of the camera.

[0174] In some embodiments, the camera pose can be determined based solely on the camera's first relative pose. For example, the camera pose of the initial frame is determined as the reference pose in the world coordinate system (e.g., the initial frame camera pose is set as the identity matrix, representing that the initial frame camera is located at the origin of the world coordinate system and facing the default direction). The first relative pose of the target frame image is combined with the camera pose of the previous frame image to obtain the absolute pose of the camera when acquiring the target frame image. Through this frame-by-frame pose recursion, the global pose of the camera when acquiring each frame in a continuous frame sequence can be obtained.

[0175] In some embodiments, a second relative pose of the camera capturing adjacent frames can be determined based on the depth map of the target frame image and the depth map of the previous frame image. The camera pose is then determined based on the first and second relative poses.

[0176] For example, point cloud reconstruction is performed on the depth map of the target frame image and the depth map of the previous frame image to obtain the sparse 3D point cloud corresponding to the target frame image and the sparse 3D point cloud corresponding to the previous frame image. The Iterative Closest Point (ICP) algorithm is used to register the sparse 3D point cloud corresponding to the target frame and the sparse 3D point cloud corresponding to the previous frame image. The transformation matrix between the two frame point clouds is obtained based on the registration result. This transformation matrix is ​​the second relative pose of the adjacent frame images captured by the camera.

[0177] After obtaining the first and second relative poses, different weights can be assigned to them based on their estimation accuracy (such as the reprojection error of the PnP algorithm and the point cloud registration error of the ICP algorithm), and weighted fusion can be performed to obtain the relative poses corresponding to adjacent frames captured by the camera.

[0178] When weighted fusion of two relative poses, if the reprojection error of the PnP algorithm is smaller, the first relative pose is given a higher weight; if the point cloud registration error of the ICP algorithm is lower, the weight of the second relative pose is increased. This weighted fusion strategy fully utilizes the advantages of both pose estimation methods, effectively reducing the error accumulation caused by a single method and further improving the accuracy of camera pose estimation. Subsequently, the fused relative pose is recursively solved by combining it with the absolute pose of the camera when acquiring the previous frame image. This yields the initial pose of the camera when acquiring the target frame image. This initial pose can be used as the initial value for subsequent optimizations, such as using the feature point cloud in the map to construct a reprojection error term for further optimization to obtain a more accurate camera pose, thus achieving accurate estimation of the front-end camera pose.

[0179] For example, camera pose determined based on a first relative pose and a second relative pose can be calibrated based on visual reprojection and loop closure detection. For instance, first, visual reprojection verification is performed by projecting map points from a 3D point cloud onto the current target frame image and matching them with feature points extracted from the image, calculating the reprojection error for each matching point. If the error exceeds a preset threshold, the point pair is marked as an anomaly and filtered. For the remaining valid point pairs, a nonlinear least squares problem with camera pose as the optimization variable is constructed to minimize the overall reprojection error, achieving local calibration of the camera pose. Next, a loop closure detection mechanism is introduced: by comparing the similarity of geometric features between the target frame image and its previous frame, a loop is determined. When a loop is detected, the associated point pair between the looped frame image and the target frame image is extracted, the relative pose constraint between the two frames is calculated, and this constraint is added to the global optimization problem. This constraint is then jointly optimized with previous pose accumulation information and reprojection constraints to eliminate the cumulative drift of long-term pose estimation and ensure the global consistency of the camera pose.

[0180] In some embodiments, pose calibration can also be performed based on an inertial measurement unit (IMU). For example... Figure 12 As shown, the third relative pose of adjacent frames captured by the camera can be determined based on the inertial data measured by the inertial measurement unit.

[0181] For example, the third relative pose can be obtained by recursively calculating the rotation and translation changes of the camera between adjacent frames through inertial integration based on the acceleration and angular velocity information collected by the IMU.

[0182] In some embodiments, the camera pose can be determined based on a first relative pose, a second relative pose, and a third relative pose.

[0183] Specifically, a first residual can be determined based on a first relative pose, a second residual can be determined based on a second relative pose, and a third residual can be determined based on a third relative pose. Then, an objective function is constructed based on the first, second, and third residuals; the target relative pose of the camera is determined based on the objective function; and the camera pose is determined based on the target relative pose of the camera and the known camera pose when the camera acquired the target frame image in the previous frame.

[0184] For example, the first residual can be a reprojection residual, the second residual can be a point cloud registration residual, and the third residual can be an inertial motion residual. The determination method of the first residual, the second residual, and the third residual can be set by those skilled in the art according to the actual situation, and the embodiments disclosed herein do not limit this.

[0185] After obtaining the first, second, and third residuals, an overall optimization objective function can be constructed based on the adjacent frame bundle adjustment method and the local keyframe bundle adjustment method, according to the first, second, and third residuals. The joint optimization of the camera pose can be achieved by minimizing the objective function.

[0186] In some embodiments, the objective function can be represented by the following formula (2).

[0187] Formula (2): .

[0188] In the above formula (2), Indicated by Find the minimum value for the camera state variables to be optimized (the camera state variables include the camera's spatial position and attitude in the world coordinate system, such as spatial three-dimensional coordinates and attitude such as three-dimensional rotation). For the visual residual term constructed based on the first residual, For image feature point index (corresponding to image feature points extracted and matched in S1120~S1130). For the first The pixel coordinates of each feature point (corresponding to the pixel coordinates of the image feature points in the target frame image extracted by the feature extraction base model in S1120). The first residual; For the ICP point cloud residual terms constructed based on the second residual, For point cloud indexing, for Frame relative The relative pose of the frame. The second residual; This refers to the IMU inertial residual term constructed based on the third residual. Indexing the IMU measurement frames. It is the third residual.

[0189] Figure 14 The interaction of three types of pose constraints—first relative pose, second relative pose, and third relative pose—is visualized to demonstrate the principle of joint optimization of camera pose through a multi-source fusion factor graph architecture.

[0190] like Figure 14 As shown, , , , ..., The term represents the camera state variables to be optimized at different times (corresponding to X in formula (2), i.e., the spatial position and attitude of the camera in the world coordinate system at different times), i.e., the values ​​in formula (2) above. . , , , ..., The symbols represent semantic features (instance-level semantic features). Light-shaded circles represent error terms (or constraint factors) arising from visual constraints corresponding to the first relative pose, dark-shaded circles represent error terms (or constraint factors) arising from geometric constraints corresponding to the second relative pose, and rectangles represent constraint factors corresponding to inertial constraints corresponding to the third relative pose. Figure 14 It can be seen that the three types of constraint factors—first relative pose, second relative pose, and third relative pose—act together on the camera state variables through a connecting structure, unifying the three heterogeneous types of information—visual, geometric, and inertial—into a single optimization framework. In this embodiment, based on the above objective function, the joint optimization of camera pose is achieved by minimizing the sum of the residuals of the three types of constraints, effectively improving the accuracy and robustness of pose estimation.

[0191] S1110~S1150 can be passed as follows Figure 1The processor 1011 in the electronic device 101 shown executes the commands, or can be executed by, for example, Figure 18 The processor 1801 shown is used for execution. Alternatively, it can be executed via, as shown in... Figure 20 The processor 111 in the electronic device 11 shown executes the commands.

[0192] like Figure 15 As shown, this disclosure also provides another embodiment, which performs the following operations S1510 to S1530.

[0193] It should be noted that S1510~S1530 together constitute a semantic query function. This function can be implemented as an independent technical solution or as an additional function of the above-mentioned positioning and mapping process (corresponding to S210~S240 and related steps).

[0194] S1510: Based on the visual language basic model and target instances in the target frame image, determine the cross-modal features representing the appearance and semantic attributes of each first instance in the 3D point cloud map.

[0195] Cross-modal features refer to feature vectors that can simultaneously characterize the visual appearance and semantic attributes of an instance. These feature vectors can reside in the same embedding space as text feature vectors, thus supporting cross-modal retrieval between text and images. For example, the CLIP (Contrastive Language-Image Pre-training) model can be used as the basic model for visual language. The instance mask region of the target instance is input into the CLIP image encoder to extract the cross-modal feature vector of the instance.

[0196] This cross-modal feature differs from the semantic features in S220 (such as IGGT features): semantic features are mainly used for similarity comparison between instances (such as semantic matching in S710~S740), while cross-modal features are mainly used to support text queries (such as a user entering "vehicles" to find vehicle instances on a map). The two types of features can coexist, each serving a different application purpose.

[0197] In one exemplary implementation, S1510 can be executed during the 3D point cloud map update process (e.g., after S240) to generate cross-modal features for each existing first instance in the map and store them in association with their corresponding instance IDs for quick retrieval when responding to subsequent text queries.

[0198] S1520: Based on the received query text and the corresponding cross-modal features of each first instance, determine the fourth instance in each 3D point cloud map that corresponds to the query text.

[0199] The fourth instance refers to an instance (which can be one or more) in the text query.

[0200] For the first, second, third, and fourth instances mentioned above, the first instance is a map instance already stored in the 3D point cloud map; the second instance is an instance detected in the target frame image within the current sliding window, used for local tracking and association; the third instance is an instance detected in the anchor frame image within the previous sliding window, used for cross-window semantic feature transfer; the second and third instances are data representations at the image detection level, used for instance association and updating during the mapping process; and the fourth instance is the data representation corresponding to the query file, used to respond to the user's query request. Any of the above instances can be information about any object such as a vehicle, pedestrian, or obstacle. It should be noted that the first, second, third, and fourth instances in this disclosure can be the same or different.

[0201] For example, the query text can be input into the text encoder of a visual language base model (such as the CLIP model) to generate a cross-modal feature vector corresponding to the text. Then, the similarity between this text's cross-modal feature vector and the cross-modal feature vectors of each first instance in the 3D point cloud map is calculated. First instances with similarity values ​​higher than a preset similarity threshold are selected as candidate instances, and the instance with the highest similarity is chosen as the fourth instance corresponding to the query text. In this way, the semantic map can support instance queries; even if the query text contains words not predefined during the map construction phase, corresponding instances can be found through cross-modal feature matching, thus facilitating user queries.

[0202] S1530: Output the three-dimensional coordinates of the fourth instance.

[0203] Each first instance in the 3D point cloud map stores its 3D position information in the world coordinate system (such as the 3D coordinates of the instance's center point or the 3D bounding box coordinates). After determining the fourth instance corresponding to the query text, the associated 3D position information of the fourth instance is extracted and output to the user in a preset format (e.g., "Instance Name: [Fourth Instance Name], 3D Coordinates: (x, y, z)" or "3D Bounding Box: (min_x, min_y, min_z, max_x, max_y, max_z)"). For example, when a user enters the query text "table" and matches the corresponding fourth instance, the system will output the table's 3D coordinates in the world coordinate system, helping the user quickly locate the target object's specific location in space, thus combining semantic map querying with spatial positioning functions.

[0204] In some examples, in addition to outputting the 3D coordinates of the fourth instance, the output can also include: the semantic category of the instance, the unique identifier (ID) of the instance, and visualization information (such as the point cloud or bounding box of the instance).

[0205] For S1520~S1530 above, the query function can lower the threshold for users to search for and locate objects in the map. Regardless of whether the user's query terms are predefined when building the map, the corresponding target instance can be quickly matched and its spatial location can be output, thereby significantly improving the flexibility and practicality of instance or target query.

[0206] It should be noted that when performing query services, the system can respond to query requests initiated by users through any calling method at any time, executing S1520 and S1530 without interrupting or waiting for the real-time 3D point cloud map update thread; when updating the map, the cross-modal feature library can be updated in real time to maintain the timeliness of the query service.

[0207] S1510~S1530 can be passed as follows Figure 1 The processor 1011 in the electronic device 101 shown executes the commands, or can be executed by, for example, Figure 18 The processor 1801 shown can be used to execute this, or it can be performed by, as shown in the example Figure 20 The processor 111 in the electronic device 11 shown executes the commands.

[0208] In some embodiments, such as Figure 16 As shown, the execution process of S1510 may include the following S1610~S1630.

[0209] S1610: Based on the visual language basic model, cross-modal feature extraction is performed on the instance mask of the target instance in the target frame image to obtain the view cross-modal features of the target instance in the target frame image.

[0210] The visual language foundation model can be a cross-modal model such as a Contrastive Language-Image Pre-training (CLIP) model. For example... Figure 17 As shown, the instance mask region corresponding to the target instance in the target frame image can be input into the CLIP model for multi-level CLIP feature extraction to obtain the cross-modal feature vector of the target instance under the current view.

[0211] S1620: Based on the cross-modal features of the target instance in the viewpoint corresponding to the target frame image, determine the first instance of the target that matches the target instance among the first instances in the 3D point cloud map.

[0212] Each first instance in the 3D point cloud map pre-stores its cross-modal feature set across multiple viewpoints. The cosine similarity between the cross-modal feature vector of the target instance and each feature vector in the cross-modal feature set of each first instance can be calculated. The maximum similarity value within this set is taken as the matching degree between the target instance and the corresponding first instance. First instances with matching degrees higher than a preset matching degree threshold are included in the candidate instance list, and the one with the highest matching degree is selected as the target first instance corresponding to the target instance.

[0213] S1630: The cross-modal features of the target instance in the target frame image are fused with the current cross-modal features of the first target instance in the 3D point cloud map to obtain the cross-modal features of the first target instance.

[0214] The method for fusing the cross-modal features of the target instance in the target frame image with the current cross-modal features of the first target instance in the 3D point cloud map can be set by those skilled in the art according to the actual situation. For example, a weighted average method can be used for fusion, and this disclosure does not limit this.

[0215] S1610~S1630 can be passed as follows Figure 1 The processor 1011 in the illustrated electronic device 101 executes the commands. Alternatively, it can be performed via, as shown... Figure 18 The processor 1801 shown can be used to execute this, or it can be performed by, as shown in the example Figure 20 The processor 111 in the electronic device 11 shown executes the commands.

[0216] Exemplary device

[0217] Figure 18 This disclosure provides a localization and mapping device based on visual semantics. For example... Figure 18 As shown, the visual semantic-based localization and mapping device 1800 includes one or more processors 1801. The processor 1801 may be a central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU), digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or other general-purpose processors, any other conventional processor, or any combination of the above processors. The processor 1801 is used to execute computer program instructions to implement the visual semantic-based localization and mapping method of this disclosure embodiment.

[0218] The visual semantic-based localization and mapping device 1800 may further include a memory 1802 for storing computer program instructions executable by the processor 1801. The memory 1802 may include volatile memory (such as random access memory, RAM) and / or non-volatile memory (such as read-only memory, flash memory, hard disk, etc.). The memory 1802 is also used to store target frame image information.

[0219] Processor 1801 reads and executes computer program instructions from memory 1802. Based on the target frame images captured by the camera (or other image acquisition device) stored in memory, it executes instructions to implement operations S210~S240, and stores the constructed 3D point cloud map in memory 1802 or outputs it to an external device as needed. Processor 1801 can be... Figure 1 The processor 1011 shown.

[0220] The processor 1801 is configured to: determine the camera pose and the depth map corresponding to the target frame image based on the target frame image acquired by the camera; determine the target instance and the semantic features of the target instance in the target frame image; determine the depth information of the target instance in the target frame image based on the depth map corresponding to the target frame image and the target instance; and update the 3D point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image.

[0221] In some embodiments, the processor 1801 executes instructions to update a 3D point cloud map based on the camera pose, depth information of the target instance in the target frame image, and semantic features of the target instance in the target frame image. This is further configured to: match the target instance with a first instance in the 3D point cloud map based on the camera pose and the instance mask of the target instance in the target frame image to determine a first matching result; and / or, match the target instance with the first instance in the 3D point cloud map based on the semantic features of the target instance in the target frame image to determine a second matching result; determine a target matching result based on the first matching result and / or the second matching result; and update the 3D point cloud map based on the target matching result, the camera pose, the depth information of the target instance in the target frame image, and the instance mask of the target instance in the target frame image.

[0222] In some embodiments, the processor 1801 executes instructions to determine a target matching result based on a first matching result and / or a second matching result, and is further configured to: determine the first matching result as the target matching result in response to the first matching result being a successful match; and determine the second matching result as the target matching result in response to the first matching result being a failed match.

[0223] In some embodiments, the processor 1801 executes instructions to determine a target matching result based on a first matching result and / or a second matching result, and is further configured to: determine the target matching result based on the first matching result and the second matching result.

[0224] In some embodiments, the processor 1801 executes instructions to implement a camera pose and an instance mask of the target instance in a target frame image, matches the target instance with a first instance in a 3D point cloud map, and determines a first matching result. This is further configured to: project each first instance in the 3D point cloud map onto the target frame image from the camera's corresponding viewpoint based on the camera pose; determine the number of first projection points of each first instance falling into the instance mask of the target instance in the target frame image; in response to the number of first projection points of each first instance being less than a first preset threshold, determine the first matching result as a matching failure; in response to the existence of one or more first instances having a first projection point count greater than or equal to the first preset threshold, determine the number of first projection points corresponding to each first instance, identify the first instance with the largest number of first projection points as the matching first instance matching the target instance, and determine the first matching result as a matching success.

[0225] In some embodiments, the processor 1801 executes instructions to match the target instance with each first instance of the 3D point cloud map based on the semantic features corresponding to the target instance in the target frame image, and determines a second matching result. This is further configured to: calculate a first similarity between the semantic features corresponding to each first instance and the target instance in the target frame image, obtaining a first similarity corresponding to each first instance; determine the maximum similarity among the first similarities corresponding to each first instance; determine the second matching result as a successful match in response to the maximum similarity being greater than or equal to a second preset threshold; and determine the second matching result as a failed match in response to the maximum similarity being less than the second preset threshold.

[0226] In some embodiments, the processor 1801 executes instructions to update a 3D point cloud map based on the target matching result, the camera pose, the depth information corresponding to the target instance in the target frame image, and the instance mask corresponding to the target instance in the target frame image. This includes: if the target matching result is successful, back-projecting the target instance onto the map coordinate system corresponding to the 3D point cloud map based on the camera pose, the depth information corresponding to the target instance in the target frame image, and the instance mask of the target instance in the target frame image, to obtain a matching instance point cloud of the target instance in the map coordinate system; fusing the matching instance point cloud with the first instance point cloud corresponding to the first matching instance in the 3D point cloud map to obtain an updated instance point cloud corresponding to the first matching instance; wherein the first matching instance is the first instance in the 3D point cloud map that matches the target instance.

[0227] In some embodiments, the processor 1801 executes instructions to update the 3D point cloud map based on the target matching result, the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image. This is further configured to: if the target matching result is a matching failure, based on the camera pose, the depth information of the target instance in the target frame image, and the instance mask of the target instance in the target frame image, back-project the target instance to the map coordinate system corresponding to the 3D point cloud map to obtain the matching instance point cloud of the target instance in the map coordinate system; and add the matching instance point cloud to the 3D point cloud map.

[0228] In some embodiments, the processor 1801 executes instructions to implement the semantic features of the target instance in the target frame image, and is further configured to: determine the identifier and semantic features of each second instance in the target frame image within the current sliding window based on a sliding window with a preset sliding step size; and update the identifier and semantic features of each second instance in the target frame image based on the identifier and semantic features of each third instance in the anchor frame image within the previous sliding window of the current sliding window.

[0229] In some embodiments, the processor 1801 executes instructions to update the identifiers and semantic features of each second instance in the target frame image based on the identifiers and semantic features of each third instance in the anchor frame image within the previous sliding window of the current sliding window, and is further configured to: calculate a second similarity between the semantic features of the third instance in the anchor frame image and the semantic features of each second instance in the target frame image within the current sliding window; and update the identifiers and semantic features of the second instance based on the identifiers and semantic features of the third instance in response to the second similarity being greater than a third preset threshold.

[0230] In some embodiments, the processor 1801 executes instructions to determine the camera pose and the depth map corresponding to the target frame image based on the target frame image acquired by the camera. This is further configured to: determine the depth map of the target frame image based on the target frame image and the intrinsic and extrinsic parameters of the camera that acquired the target frame image; perform feature extraction and feature association on the target frame image and the previous frame image to obtain multiple pairs of associated points in adjacent frames; determine the three-dimensional image feature points corresponding to the target frame image and the three-dimensional image feature points corresponding to the previous frame image based on the multiple pairs of associated points and the camera's intrinsic and extrinsic parameters; determine the first relative pose of the camera capturing adjacent frame images based on the multiple pairs of associated points, the three-dimensional image feature points corresponding to the target frame image, and the three-dimensional image feature points corresponding to the previous frame image; and determine the camera pose of each camera based on the first relative pose of the cameras.

[0231] In some embodiments, the processor 1801 executes instructions to perform feature extraction and feature association on the target frame image captured by the camera and the previous frame image captured by the camera to obtain multiple associated point pairs in adjacent frames. This is further configured to: extract image feature points of the target frame image and the previous frame image based on a feature extraction basic model; and perform feature association processing on the image feature points of the target frame image and the previous frame image based on a feature matching basic model to obtain multiple associated point pairs.

[0232] In some embodiments, the processor 1801 executes instructions to determine the camera pose based on the first relative pose, and is further configured to: determine the second relative pose of the camera capturing adjacent frame images based on the depth map of the target frame image and the depth map of the previous frame image of the target frame image; determine the third relative pose of the camera capturing adjacent frame images based on inertial data measured by the inertial measurement unit; and determine the camera pose based on the first relative pose, the second relative pose, and the third relative pose.

[0233] In other embodiments, such as Figure 19 As shown, the processor 1801 in the visual semantic-based localization and mapping device 1800 may include a first processing unit 18011 and a second processing unit 18012. The first processing unit 18011 is used to implement the 3D point cloud map update function, as in S210-S240; the second processing unit 18012 can be used to implement query services.

[0234] In some embodiments, the processor 1801 is further configured to: after determining the target instance in the target frame image, determine cross-modal features representing the appearance and semantic attributes of each first instance in the 3D point cloud map based on the visual language base model and the target instance in the target frame image; determine a fourth instance in each 3D point cloud map corresponding to the query text based on the received query text and the corresponding cross-modal features of each first instance; and output the 3D coordinates of the fourth instance.

[0235] In some embodiments, the processor 1801 executes instructions to determine cross-modal features representing the appearance and semantic attributes of each first instance in a 3D point cloud map based on a visual language base model and a target instance in the target frame image. This is further configured to: determine cross-modal features representing the appearance and semantic attributes of each first instance in a 3D point cloud map based on a visual language base model and a target instance in the target frame image, including: extracting cross-modal features from the instance mask of the target instance in the target frame image based on the visual language base model to obtain the viewpoint cross-modal features of the target instance in the target frame image; determining a target first instance matching the target instance among the first instances in the 3D point cloud map based on the viewpoint cross-modal features of the target instance in the target frame image; and fusing the viewpoint cross-modal features of the target instance in the target frame image with the current cross-modal features of the target first instance in the 3D point cloud map to obtain the cross-modal features of the target first instance.

[0236] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects in the exemplary method section above, and will not be repeated here.

[0237] Exemplary electronic devices

[0238] Figure 20 A structural diagram of an electronic device provided in this disclosure includes one or more processors 111 and a memory 112.

[0239] The processor 111 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 11 to perform desired functions.

[0240] The memory 112 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 111 may execute one or more computer program instructions to implement the visual semantic-based localization and mapping methods and / or other desired functions of the various embodiments of this disclosure described above.

[0241] In one example, the electronic device 11 may also include an input device 113 and an output device 114, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0242] The input device 113 may also include, for example, a keyboard, a mouse, etc.

[0243] The output device 114 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0244] Of course, for the sake of simplicity, Figure 20 Only some of the components of the electronic device 11 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 11 may include any other suitable components depending on the specific application.

[0245] Exemplary computer program products and computer-readable storage media

[0246] In addition to the methods and devices described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the visual semantic-based localization and mapping methods of various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0247] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0248] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps of the visual semantic-based localization and mapping methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0249] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0250] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0251] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A localization and mapping method based on visual semantics, comprising: Based on the target frame image captured by the camera, determine the camera pose of the camera and the depth map corresponding to the target frame image; Determine the target instance in the target frame image and the semantic features of the target instance in the target frame image; Based on the depth map corresponding to the target frame image and the target instance, the depth information of the target instance in the target frame image is determined; The 3D point cloud map is updated based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image.

2. The method according to claim 1, wherein, The update of the 3D point cloud map based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image includes: Based on the camera pose of the camera and the instance mask of the target instance in the target frame image, the target instance is matched with a first instance of the 3D point cloud map to determine a first matching result; and / or, based on the semantic features of the target instance in the target frame image, the target instance is matched with the first instance of the 3D point cloud map to determine a second matching result. Based on the first matching result and / or the second matching result, determine the target matching result; The 3D point cloud map is updated based on the target matching result, the camera pose, the depth information of the target instance in the target frame image, and the instance mask of the target instance in the target frame image.

3. The method according to claim 2, wherein, Determining the target matching result based on the first matching result and / or the second matching result includes: In response to the first matching result being a successful match, the first matching result is determined as the target matching result; In response to the first matching result being a failed match, the second matching result is determined as the target matching result.

4. The method according to claim 2, wherein, Determining the target matching result based on the first matching result and / or the second matching result includes: The target matching result is determined based on the first matching result and the second matching result.

5. The method according to claim 2, wherein, The step of matching the target instance with a first instance of the 3D point cloud map based on the camera pose and the instance mask of the target instance in the target frame image, and determining a first matching result, includes: Based on the camera pose of the camera, each first instance in the 3D point cloud map is projected onto the target frame image under the corresponding view of the camera, and the number of first projection points of each first instance falling into the instance mask of the target instance in the target frame image is determined. In response to the fact that the number of first projection points in each of the first instances is less than a first preset threshold, the first matching result is determined to be a matching failure. In response to the existence of one or more instances where the number of first projection points is greater than or equal to the first preset threshold, the number of first projection points corresponding to each first instance in each first instance is determined, the first instance with the largest number of first projection points is determined as the matching first instance that matches the target instance, and the first matching result is determined to be a successful match.

6. The method according to claim 2, wherein, The step of matching the target instance with each first instance of the 3D point cloud map based on the semantic features corresponding to the target instance in the target frame image to determine the second matching result includes: Calculate the first similarity of the semantic features corresponding to each first instance and the target instance in the target frame image to obtain the first similarity corresponding to each first instance; Determine the maximum similarity among the first similarities corresponding to each of the first instances; In response to the maximum similarity being greater than or equal to a second preset threshold, the second matching result is determined to be a successful match; In response to the maximum similarity being less than the second preset threshold, the second matching result is determined to be a matching failure.

7. The method according to claim 2, wherein, The step of updating the 3D point cloud map based on the target matching result, the camera pose of the camera, the depth information corresponding to the target instance in the target frame image, and the instance mask corresponding to the target instance in the target frame image includes: If the target matching result is a successful match, based on the camera pose of the camera, the depth information corresponding to the target instance in the target frame image, and the instance mask of the target instance in the target frame image, the target instance is back-projected onto the map coordinate system corresponding to the 3D point cloud map to obtain the matching instance point cloud of the target instance in the map coordinate system. The matching instance point cloud is fused with the first instance point cloud corresponding to the first matching instance in the 3D point cloud map to obtain the updated instance point cloud corresponding to the first matching instance; wherein, the first matching instance is the first instance in the 3D point cloud map that matches the target instance.

8. The method according to claim 2, wherein, The update of the 3D point cloud map based on the target matching result, the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image includes: If the target matching result is a failure, based on the camera pose of the camera, the depth information corresponding to the target instance in the target frame image, and the instance mask of the target instance in the target frame image, the target instance is back-projected onto the map coordinate system corresponding to the 3D point cloud map to obtain the matching instance point cloud of the target instance in the map coordinate system. Add the matching instance point cloud to the 3D point cloud map.

9. The method according to claim 1, wherein, Determining the semantic features of the target instance in the target frame image includes: Based on a sliding window with a preset sliding step size, the identifier and semantic features of each second instance in the target frame image within the current sliding window are determined. Based on the identifiers and semantic features of each third instance in the anchor frame image of the previous sliding window of the current sliding window, update the identifiers and semantic features of each second instance in the target frame image.

10. The method according to claim 9, wherein, The step of updating the identifiers and semantic features of each second instance in the target frame image based on the identifiers and semantic features of each third instance in the anchor frame image of the previous sliding window of the current sliding window includes: Calculate the second similarity between the semantic features of the third instance in the anchor frame image and the semantic features of the second instance in each target frame image within the current sliding window; In response to the second similarity being greater than a third preset threshold, the identifier and semantic features of the second instance are updated based on the identifier and semantic features of the third instance, respectively.

11. The method according to claim 1, wherein, The step of determining the camera pose and the corresponding depth map of the target frame image based on the target frame image acquired by the camera includes: Based on the target frame image and the intrinsic and extrinsic parameters of the camera that acquired the target frame image, the depth map of the target frame image is determined; Feature extraction and feature association are performed on the target frame image and the previous frame image of the target frame image to obtain multiple pairs of associated points in adjacent frames; Based on the multiple associated point pairs and the intrinsic and extrinsic parameters of the camera, the three-dimensional image feature points corresponding to the target frame image and the three-dimensional image feature points corresponding to the previous frame image of the target frame image are determined. Based on the multiple associated point pairs, the three-dimensional image feature points corresponding to the target frame image, and the three-dimensional image feature points corresponding to the previous frame image of the target frame image, the first relative pose of the camera capturing adjacent frame images is determined; Based on the first relative pose of the cameras, the camera pose of each camera is determined.

12. The method according to claim 11, wherein, The step involves extracting and associating features between the target frame image captured by the camera and the previous frame image captured by the camera to obtain multiple pairs of associated points in adjacent frames, including: Based on the feature extraction model, image feature points of the target frame image and image feature points of the previous frame image of the target frame image are extracted; Based on the feature matching model, feature association processing is performed on the image feature points of the target frame image and the image feature points of the previous frame image to obtain the multiple associated point pairs.

13. The method according to claim 11 or 12, wherein, Determining the camera pose based on the first relative pose includes: Based on the depth map of the target frame image and the depth map of the previous frame image of the target frame image, the second relative pose of the camera capturing the adjacent frame images is determined; Based on the inertial data measured by the inertial measurement unit, the third relative pose of the adjacent frame images captured by the camera is determined; The camera pose is determined based on the first relative pose, the second relative pose, and the third relative pose.

14. The method according to claim 1, further comprising, after determining the target instance in the target frame image: Based on the visual language basic model and the target instance in the target frame image, determine the cross-modal features representing the appearance and semantic attributes of each first instance in the 3D point cloud map; Based on the received query text and the corresponding cross-modal features of each of the first instances, a fourth instance corresponding to the query text is determined in each of the three-dimensional point cloud maps; Output the three-dimensional coordinates corresponding to the fourth instance.

15. The method according to claim 14, wherein, The method of determining cross-modal features representing the appearance and semantic attributes of each first instance in the 3D point cloud map based on the visual language basic model and the target instance in the target frame image includes: Based on the visual language basic model, cross-modal feature extraction is performed on the instance mask of the target instance in the target frame image to obtain the view cross-modal features of the target instance in the target frame image. Based on the cross-modal features of the viewpoint corresponding to the target instance in the target frame image, a first target instance matching the target instance is determined among each first instance in the 3D point cloud map; The cross-modal features of the target instance corresponding to the target frame image are fused with the current cross-modal features of the first target instance in the 3D point cloud map to obtain the cross-modal features of the first target instance.

16. A localization and mapping device based on visual semantics, comprising a processor, the processor being configured to: Based on the target frame image captured by the camera, determine the camera pose of the camera and the depth map corresponding to the target frame image; Determine the target instance in the target frame image and the semantic features of the target instance in the target frame image; Based on the depth map corresponding to the target frame image and the target instance, the depth information of the target instance in the target frame image is determined; The 3D point cloud map is updated based on the camera pose, the depth information of the target instance in the target frame image, and the semantic features of the target instance in the target frame image.

17. The apparatus according to claim 16, wherein, The processor is further configured to: Based on the camera pose of the camera and the instance mask of the target instance in the target frame image, the target instance is matched with a first instance of the 3D point cloud map to determine a first matching result; and / or, based on the semantic features of the target instance in the target frame image, the target instance is matched with the first instance of the 3D point cloud map to determine a second matching result. Based on the first matching result and / or the second matching result, determine the target matching result; The 3D point cloud map is updated based on the target matching result, the camera pose, the depth information of the target instance in the target frame image, and the instance mask of the target instance in the target frame image.

18. The apparatus according to claim 17, wherein, The processor is further configured to: In response to the first matching result being a successful match, the first matching result is determined as the target matching result; In response to the first matching result being a failed match, the second matching result is determined as the target matching result.

19. The apparatus according to claim 17, wherein, The processor is further configured to: The target matching result is determined based on the first matching result and the second matching result.

20. The apparatus according to claim 17, wherein, The processor is further configured to: Based on the camera pose of the camera, each first instance in the 3D point cloud map is projected onto the target frame image from the corresponding viewpoint of the camera; Determine the number of first projection points of each first instance that fall into the instance mask of the target instance in the target frame image; In response to the fact that the number of first projection points in each of the first instances is less than a first preset threshold, the first matching result is determined to be a matching failure. In response to the existence of one or more instances where the number of first projection points is greater than or equal to the first preset threshold, the number of first projection points corresponding to each first instance in each first instance is determined, the first instance with the largest number of first projection points is determined as the matching first instance that matches the target instance, and the first matching result is determined to be a successful match.

21. The apparatus according to claim 17, wherein, The processor is further configured to: Calculate the first similarity of the semantic features corresponding to each first instance and the target instance in the target frame image to obtain the first similarity corresponding to each first instance; Determine the maximum similarity among the first similarities corresponding to each of the first instances; In response to the maximum similarity being greater than or equal to a second preset threshold, the second matching result is determined to be a successful match; In response to the maximum similarity being less than the second preset threshold, the second matching result is determined to be a matching failure.

22. The apparatus according to claim 17, wherein, The processor is further configured to: If the target matching result is a successful match, based on the camera pose of the camera, the depth information corresponding to the target instance in the target frame image, and the instance mask of the target instance in the target frame image, the target instance is back-projected onto the map coordinate system corresponding to the 3D point cloud map to obtain the matching instance point cloud of the target instance in the map coordinate system. The matching instance point cloud is fused with the first instance point cloud corresponding to the first matching instance in the 3D point cloud map to obtain the updated instance point cloud corresponding to the first matching instance; wherein, the first matching instance is the first instance in the 3D point cloud map that matches the target instance.

23. The apparatus according to claim 17, wherein, The processor is further configured to: If the target matching result is a failure, based on the camera pose of the camera, the depth information corresponding to the target instance in the target frame image, and the instance mask of the target instance in the target frame image, the target instance is back-projected onto the map coordinate system corresponding to the 3D point cloud map to obtain the matching instance point cloud of the target instance in the map coordinate system. Add the matching instance point cloud to the 3D point cloud map.

24. The apparatus according to claim 16, wherein, The processor is further configured to: Based on a sliding window with a preset sliding step size, the identifier and semantic features of each second instance in the target frame image within the current sliding window are determined. Based on the identifiers and semantic features of each third instance in the anchor frame image of the previous sliding window of the current sliding window, update the identifiers and semantic features of each second instance in the target frame image.

25. The apparatus according to claim 24, wherein, The processor is further configured to: Calculate the second similarity between the semantic features of the third instance in the anchor frame image and the semantic features of the second instance in each target frame image within the current sliding window; In response to the second similarity being greater than a third preset threshold, the identifier and semantic features of the second instance are updated based on the identifier and semantic features of the third instance, respectively.

26. The apparatus according to claim 16, wherein, The processor is further configured to: Based on the target frame image and the intrinsic and extrinsic parameters of the camera that acquired the target frame image, the depth map of the target frame image is determined; Feature extraction and feature association are performed on the target frame image and the previous frame image of the target frame image to obtain multiple pairs of associated points in adjacent frames; Based on the multiple associated point pairs and the intrinsic and extrinsic parameters of the camera, the three-dimensional image feature points corresponding to the target frame image and the three-dimensional image feature points corresponding to the previous frame image of the target frame image are determined. Based on the multiple associated point pairs, the three-dimensional image feature points corresponding to the target frame image, and the three-dimensional image feature points corresponding to the previous frame image of the target frame image, the first relative pose of the camera capturing adjacent frame images is determined; Based on the first relative pose of the cameras, the camera pose of each camera is determined.

27. The apparatus according to claim 26, wherein, The processor is further configured to: Based on the feature extraction model, image feature points of the target frame image and image feature points of the previous frame image of the target frame image are extracted; Based on the feature matching model, feature association processing is performed on the image feature points of the target frame image and the image feature points of the previous frame image to obtain the multiple associated point pairs.

28. The apparatus according to claim 26 or 27, wherein, The processor is further configured to: Based on the depth map of the target frame image and the depth map of the previous frame image of the target frame image, the second relative pose of the camera capturing the adjacent frame images is determined; Based on the inertial data measured by the inertial measurement unit, the third relative pose of the adjacent frame images captured by the camera is determined; The camera pose is determined based on the first relative pose, the second relative pose, and the third relative pose.

29. The apparatus of claim 16, wherein the processor is further configured to: Based on the visual language basic model and the target instance in the target frame image, determine the cross-modal features representing the appearance and semantic attributes of each first instance in the 3D point cloud map; Based on the received query text and the corresponding cross-modal features of each of the first instances, a fourth instance corresponding to the query text is determined in each of the three-dimensional point cloud maps; Output the three-dimensional coordinates corresponding to the fourth instance.

30. The apparatus according to claim 29, wherein, The processor is further configured to: Based on the visual language basic model, cross-modal feature extraction is performed on the instance mask of the target instance in the target frame image to obtain the view cross-modal features of the target instance in the target frame image. Based on the cross-modal features of the viewpoint corresponding to the target instance in the target frame image, a first target instance matching the target instance is determined among each first instance in the 3D point cloud map; The cross-modal features of the target instance corresponding to the target frame image are fused with the current cross-modal features of the first target instance in the 3D point cloud map to obtain the cross-modal features of the first target instance.

31. A computer-readable storage medium storing a computer program for executing the visual semantic-based localization and mapping method according to any one of claims 1-15.

32. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the visual semantic-based localization and mapping method according to any one of claims 1-15.

Images