A visual odometry positioning correction method based on environmental objects as semantic landmarks

Abstract

The application discloses a kind of visual odometry positioning correction methods based on environmental object as semantic landmark, belong to visual positioning field, this method constructs semantic map by semantic object in actual scene, and trains the deep learning model for semantic segmentation and classification;By the actual scene image collected during the operation of positioning system is executed first semantic object semantic segmentation detection and classification detection, and it is used as semantic landmark in from semantic map retrieval matching item, and when matching item is in the range of camera field of view, the pixel coordinates of its feature points are used to carry out the positioning correction of visual odometry;The method is based on semantic segmentation model, classification model and map matching and is positioned to visual odometry, using the anti-interference of semantic feature in environment, can get rid of the application limitation of visual odometry in complex indoor environment, improve the robustness and autonomy of positioning system in complex environment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of visual positioning, and more particularly relates to a visual odometer positioning correction method based on environment objects as semantic landmarks. BACKGROUND

[0002] Autonomous positioning is a core technical problem in the fields of robot navigation, unmanned driving, etc. Autonomous positioning generally refers to a technology for estimating and positioning the position and attitude without relying on external signals or information sources, through internal sensors and algorithms. Unlike traditional positioning methods relying on satellite navigation (such as GPS), autonomous positioning technology can operate in an environment without external positioning signals, wherein the odometer serves as a module for estimating device pose information from sensor data, and the accuracy and robustness of the odometer play a decisive role in the performance of the autonomous positioning system.

[0003] In traditional odometers, wheeled odometers calculate the displacement of a vehicle or robot by measuring the rotation of the wheels, which is prone to drift due to wheel slippage; precise laser radar sensors are expensive; inertial sensors calculate position and attitude by integrating acceleration and angular velocity, and measurement errors will gradually accumulate and drift over time. Visual odometers obtain environmental information through cameras and estimate the motion trajectory of a device or robot in space using changes in image feature points, and have the advantages of low cost and high-precision robust positioning in complex environments. However, in indoor low-texture environments, visual odometers use bottom-level brightness relationships for feature description and matching, which lacks anti-interference ability and is easily affected by factors such as changes in light, changes in viewing angle, and dynamic objects, resulting in feature point matching errors and cumulative errors in subsequent pose estimation, thereby affecting the final positioning result. SUMMARY

[0004] To address the problem of degraded positioning performance of visual odometers in indoor complex environments due to factors such as changes in light, scarcity of features, and limited field of view, the present application proposes a visual positioning correction method based on environment objects as semantic landmarks, which aims to correct the positioning results of visual odometers through semantic detection networks and map matching, to overcome the limitations of visual odometers in indoor environments, and ultimately to improve the robustness and autonomy of the positioning system in complex environments.

[0005] To achieve the above-mentioned purpose, according to a first aspect of the present application, a visual odometer positioning correction method based on environment objects as semantic landmarks is provided, comprising:

[0006] S1, selecting semantic objects from an actual scene, taking a world coordinate system of the actual scene as a global coordinate system, recording semantic labels and global coordinates of feature points of each semantic object to construct a semantic map; wherein the semantic object has a rectangular plane feature, and the feature points are four vertices of the rectangular plane; the semantic object includes a first semantic object, and the first semantic object has uniqueness in the actual scene;

[0007] S2, correcting a camera pose according to the semantic map, comprising:

[0008] S21, inputting a current actual scene image collected by a camera into a pre-trained semantic segmentation model for semantic segmentation to obtain a first semantic object image, and inputting the first semantic object image into a pre-trained classification model to obtain a semantic label of the first semantic object;

[0009] S22, obtaining global coordinates of feature points of the first semantic object from the semantic map according to the semantic label, and converting the global coordinates to a camera coordinate system according to a current pose of the camera provided by a visual odometry to determine whether the feature points are within a current field of view of the camera, if yes, entering S23, otherwise returning to S22;

[0010] S23, performing feature point matching on the feature points of the first semantic object according to the global coordinates and pixel coordinates of the feature points, calculating a pose of the camera using a PnP algorithm according to a matching result, and adding the pose as a factor to a factor graph to correct the pose of the camera.

[0011] According to a second aspect of the present application, an electronic device is provided, comprising: a computer readable storage medium and a processor;

[0012] The computer readable storage medium is configured to store executable instructions;

[0013] The processor is configured to read the executable instructions stored in the computer readable storage medium, and execute the method according to the first aspect.

[0014] According to a third aspect of the present application, a computer readable storage medium is provided, the computer readable storage medium stores computer instructions, and the computer instructions are configured to make the processor execute the method according to the first aspect.

[0015] According to a fourth aspect of the present application, a computer program product is provided, comprising a computer program or instructions, and the computer program or instructions are executed by the processor to implement the method according to the first aspect.

[0016] Overall, compared with the prior art, the above technical solutions conceived by the present application can achieve the following beneficial effects:

[0017] The method provided by the present application constructs a semantic map through semantic objects in an actual scene, and trains a deep learning model for semantic segmentation and classification; performs semantic segmentation detection and classification detection of a first semantic object on an actual scene image collected during operation of a positioning system, and uses the actual scene image as a semantic landmark to search for a matching item from the semantic map, and when the matching item is within the field of view of the camera, uses the pixel coordinates of the feature points thereof to perform positioning correction of visual odometry; the method corrects the positioning of visual odometry based on the matching of the semantic segmentation model, the classification model and the map, and uses the anti-interference of semantic features in the environment to break away from the application limitations of visual odometry in a complex indoor environment, and improves the robustness and autonomy of the positioning system in a complex environment.

[0018] As a further preferred embodiment of the present application, the method provided by the present application further performs semantic segmentation detection of a second semantic object on an actual scene image collected during operation of a positioning system, and searches for a plurality of second semantic objects closest to the camera from the semantic map as matching items according to the current pose provided by visual odometry, and when the matching items are within the field of view of the camera, uses the pixel coordinates of the feature points thereof to perform positioning correction of visual odometry, thereby further improving the correction accuracy of the camera pose. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A flowchart of a visual positioning correction method based on environmental objects as semantic landmarks provided by an embodiment of the present application is shown in FIG. 1.

[0020] Figure 2 An actual scene and a first semantic object therein are shown in FIG. 2.

[0021] Figure 3 An actual scene and a second semantic object therein are shown in FIG. 3.

[0022] Figure 4 A semantic map is shown in FIG. 4.

[0023] Figure 5 A feature point extraction result of a unique semantic object is shown in FIG. 5.

[0024] Figure 6 A feature point extraction result of a repeated semantic object is shown in FIG. 6.

[0025] Figure 7 A comparison chart of a single round-trip correction experiment using the method provided by an embodiment of the present application and an uncorrected result is shown in FIG. 7.

[0026] Figure 8 A comparison chart of a multiple round-trip correction experiment using the method provided by an embodiment of the present application and an uncorrected result is shown in FIG. 8.

[0027] Figure 9 A result graph of a z-axis in a global coordinate system for a plurality of times of round-trip correction experiments by the method provided by the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0029] The embodiment of the present application provides a visual odometer positioning correction method based on environmental objects as semantic landmarks, as shown in Figure 1 , which comprises the following steps.

[0030] S1, selecting semantic objects from an actual scene and taking the semantic objects as semantic landmarks, taking a world coordinate system of the actual scene as a global coordinate system, recording semantic labels of the semantic objects and global coordinates of feature points of the semantic objects, and constructing a semantic map;

[0031] The semantic objects have a rectangular plane feature, and the feature points are four vertices of the rectangular plane. The semantic objects include a first semantic object, and the first semantic object has uniqueness in the actual scene.

[0032] Specifically, a world coordinate system of an actual scene is established, and objects (i.e. semantic objects) with semantic value in the actual scene are selected as semantic landmarks. Four feature points are selected for each semantic object, and global coordinates of the feature points of the semantic objects and corresponding semantic labels are recorded to obtain a semantic map.

[0033] In order to initialize the positioning system, the semantic objects must include a first semantic object, and the first semantic object has uniqueness in the actual scene. The uniqueness means that there is only one first semantic object in a given environment, as shown in Figure 2 a doorplate number of a room. The feature points of the doorplate number are four vertices of the doorplate.

[0034] In order to increase the correction accuracy of the camera pose, as a further preferred scheme of the present application, the semantic objects can further include a second semantic object, and the second semantic object does not have uniqueness in the actual scene, i.e. the second semantic object will repeatedly appear in the same scene. As shown in Figure 2 a fire extinguisher box that repeatedly appears at certain intervals.

[0035] In addition, the semantic objects should have a plane feature, as shown in Figures 2-3The house number and the fire extinguisher shown both have planar features; correspondingly, the feature points of the semantic objects are the four vertices of their rectangular planes.

[0036] A semantic map and a world coordinate diagram established according to an actual scene are shown in FIG. 2, which includes a first semantic object, a house number, and a second semantic object, a fire extinguisher box. Figure 4

[0037] S2, correcting the camera pose according to the semantic map, comprising:

[0038] S21, inputting an actual scene image collected by the camera in real time during running of the autonomous positioning system into a pre-trained semantic segmentation model to perform semantic segmentation to obtain a first semantic object image segmented from the current actual scene image, and inputting the first semantic object image into a pre-trained classification model to obtain a semantic label of the first semantic object;

[0039] S22, obtaining global coordinates of feature points of the first semantic object from the semantic map according to the semantic label, and converting the feature points of the first semantic object from a global coordinate system to a camera coordinate system according to a current pose of the camera to determine whether the feature points are within a current field of view of the camera, if yes, entering S23, otherwise returning to S22;

[0040] S23, performing feature point matching on the feature points of the first semantic object according to the global coordinates and pixel coordinates (i.e. coordinates in a pixel coordinate system of the current actual scene image) of the feature points, calculating a pose of the camera using a PnP algorithm according to a matching result, and adding the pose as a factor into a factor graph to correct the pose of the camera.

[0041] When the semantic object further includes a second semantic object, step S2 further includes:

[0042] S21', inputting the current actual scene image into a pre-trained semantic segmentation model to perform semantic segmentation to obtain a second semantic object image;

[0043] S22', searching for N feature points of the second semantic object closest to the camera in the semantic map according to a current global pose of the camera, and converting the feature points of the second semantic object to a camera coordinate system according to the pose of the camera to determine whether the second semantic object is within a current field of view of the camera, if at least one second semantic object is within the current field of view of the camera, entering S23', otherwise returning to S22';

[0044] ​S23', according to the global coordinates and pixel coordinates of the feature points of the second semantic object, performing feature point matching on the feature points, calculating the pose of the camera using a PnP algorithm according to a matching result, and adding the pose of the camera as a factor into a factor graph to correct the pose of the camera.

[0045] The training process of the classification model of the semantic segmentation model is as follows:

[0046] The semantic segmentation model is obtained by training a first convolutional neural network using a first data set; the first data set includes actual scene images collected from multiple perspectives and first semantic object images therein.

[0047] It can be understood that, to implement the segmentation of the semantic segmentation model on the second semantic object image in S21', when the semantic object further includes a second semantic object, the training data set of the semantic segmentation model further includes second semantic object images corresponding to the actual scene images collected from multiple perspectives.

[0048] The classification model is obtained by training a second convolutional neural network using a second data set; the second data set includes the first semantic object images and semantic labels thereof in the first data set.

[0049] That is, scene image data with selected semantic objects is captured from multiple perspectives, and the data set is used to train the semantic segmentation model and the classification model:

[0050] First, image data required for training is collected and prepared for training the semantic segmentation model. The segmentation result output by the trained segmentation model is used to save and archive the segmented region, and a segmentation region image is obtained; then, the above-mentioned archived segmentation result is used to train the classification model. It is worth noting that only a unique object participates in the training process of the classification model.

[0051] The first and second convolutional neural networks can adopt any convolutional neural network. Considering the requirements of calculation accuracy and light weight, the first and second convolutional neural networks are preferably YOLO networks.

[0052] Taking the first and second convolutional neural networks as YOLO networks as an example, a real scene video is recorded and pictures of the real scene that can be used are screened out from the video, and then a marking software is used to mark the house numbers (i.e. the first semantic object) in the pictures. After the data set is prepared, the first YOLO network is trained to obtain a semantic segmentation network to detect the house number images. After the house number images are detected, the corresponding house numbers need to be obtained, so the house number image data set (i.e. the first semantic object image) and the corresponding house numbers (i.e. the semantic labels of the first semantic object image) are used to train the second YOLO network to obtain a classification model, i.e. the house number images detected by the semantic segmentation network are cut out first, and then the house numbers of different house number images are taken as different categories to train the second YOLO network to obtain the classification model.

[0053] When the semantic object further includes a fire extinguishing box (i.e. the second semantic object), the training data set of the semantic segmentation model further includes fire extinguishing box images (i.e. the second semantic object images) corresponding to the actual scene images collected from multiple perspectives.

[0054] In step S2, during the running of the visual odometer, scene images are collected in real time by the camera and semantic object segmentation detection and classification detection are performed thereon, the global pose information provided by the visual odometer is used to search for a matching item in the semantic map, when the matching is successful, feature point matching is performed according to the pixel coordinates and global coordinates of the feature points of the semantic object, so as to correct the camera pose, thereby realizing the positioning correction of the visual odometer.

[0055] It is worth noting that the above process implements differential processing according to the category of the semantic object:

[0056] (1) If the semantic object detected by the semantic segmentation has uniqueness in the actual scene, i.e. the detected semantic object is the first semantic object, the matching item of the first semantic object in the semantic map is searched according to the current global pose and the output result of the classification model:

[0057] The global coordinates of the feature points of the first semantic object in the semantic map are searched according to the semantic label of the first semantic object, the global coordinates of the feature points of the first semantic object are converted to the camera coordinate system according to the current pose of the camera and the camera imaging principle, and it is judged whether the first semantic object is within the field of view of the camera.

[0058] That is, at this time, the first semantic object is a semantic landmark, and the global coordinates of the feature points of the semantic landmark are obtained and converted to the camera coordinates, and if the coordinates x c , y cOr zc exceeds the current field of view boundary of the camera, it is considered that the first semantic object observed by the current camera is not the selected object in the semantic map, that is, the output result of the classification model does not match the actual first semantic object, and cannot be used for visual positioning correction, and then the above processing steps of collecting, segmenting and detecting the actual scene image are returned; otherwise, it can be used for visual positioning correction: the pixel coordinates and global coordinates of the feature points of the first semantic object are matched, the matching results of the pixel coordinates and global coordinates of the feature points are calculated by using the PnP algorithm, the position of the camera is calculated by using the PnP algorithm, the position of the camera is used as a factor, and the factor is added in the factor graph to realize the pose correction.

[0059] Wherein, when performing feature point matching, a unified sorting strategy is adopted, that is, the feature points under the pixel coordinates and the saved feature point coordinates are sorted in a clockwise direction, and the upper left pixel point is taken as the first reference point. In this way, the accurate matching of the pixel coordinates and the global coordinates of the feature points is ensured.

[0060] The camera imaging principle is:

[0061]

[0062] Wherein, x w ,y w ,z w is the coordinate of any point in the global coordinate system, R, t is the rotation and translation of the current pose of the camera, dx and dy respectively represent the physical size of each pixel on the horizontal axis x and the vertical axis y, f is the focal length, u0, v0 is the principal point coordinate, and u, v is the pixel coordinate.

[0063] The calculation formula for converting from the global coordinate to the camera coordinate is

[0064]

[0065] Wherein, x c ,y c ,z c is the coordinate in the camera coordinate system.

[0066] (2) If the semantic object detected by semantic segmentation does not have uniqueness in the actual scene, that is, the detected semantic object is the second semantic object, then according to the current global pose and the output result of the semantic segmentation model, the matching item of the second semantic object in the semantic map is found:

[0067] At this time, the output information of the non-classification network, only the output information of the segmentation network, that is, the image of the second semantic object, so according to the current global pose provided by the visual odometer, select the N (N>1) second semantic objects closest to the camera in the semantic map, convert the global coordinates of the feature points of each second semantic object to the camera coordinate system (conversion related calculation is the same as above), and judge one by one whether it is within the camera field of view, if at least one second semantic object is within the camera field of view, then the second semantic object observed by the current camera is the object selected in the semantic map, which can be used for visual positioning correction: using the pixel coordinates and global coordinates of the feature points of the second semantic object to match the feature points, using the matching results of the pixel coordinates and global coordinates of the feature points to calculate the pose of the camera using PnP algorithm, taking the position of the camera calculated by the PnP algorithm as a factor, and adding a factor in the factor graph to realize pose correction; Otherwise, the second semantic object observed by the current camera is not the object selected in the semantic map, that is, the output result of the classification model does not match the actual second semantic object, which cannot be used for visual positioning correction, then return to the above processing steps of collecting and segmenting the actual scene image. That is, in the case that the semantic object detected by semantic segmentation also includes the second semantic object, and the second semantic object observed by the current camera is the object selected in the semantic map, the second semantic object is used to perform pose correction again, thereby further improving the correction accuracy.

[0068] For example, select the 5 fire extinguisher boxes closest to the current pose in the semantic map, and judge one by one whether they are within the field of view, if at least one fire extinguisher box is within the camera field of view, then the pixel coordinates of its feature points can be used for visual positioning correction.

[0069] The pixel coordinates of the feature points of the first and second semantic objects can be calculated by existing methods. Considering that the semantic object has a rectangular plane feature, the geometric property of the semantic object may be a plane or a solid. To improve the calculation accuracy, as a further preferred embodiment of the present application:

[0070] If the geometric property of the semantic object is a plane, the pixel coordinates of the feature points are determined as follows: extract all corner points in the semantic object image, filter out the corner points whose distance to the boundary of the semantic object image is less than the distance threshold, and take the pixel coordinates of the four corner points corresponding to the four vertices of the semantic object image respectively as the pixel coordinates of the feature points of the semantic object.

[0071] That is, if the detected object belongs to the plane type, the pixel coordinates of the feature points of the semantic object are determined by using point feature selection technology. For example, Figure 5As shown in the figure, the rectangular bounding box of the house number is extracted from the output of the segmentation model, the image is converted into grayscale mode, and the area within the bounding box is cropped to obtain the house number image. Subsequently, the corner detection algorithm is applied to extract corner points from the cropped image. The corner points whose distance from the boundary of the house number image is less than the distance threshold are preliminarily screened out. The distance threshold is S / K+b, where S is the area of ​​the area, K is the scale factor, and b is the bias factor. All of them can be set according to experience, that is, the larger the area of ​​the area, the larger the distance threshold. Finally, based on the distance between the feature point and the vertices of the rectangle (i.e., the vertices of the bounding box), the four corner points with the smallest distance are screened, which are the four feature points of the house number, and their pixel coordinates are the pixel coordinates of the four feature points of the house number.

[0072] If the geometric attribute of the semantic object is three-dimensional, the pixel coordinates of its feature points are determined as follows: identify all straight lines in the semantic object image, group straight lines with equal slopes into the same group, preprocess each group of straight lines, extend the preprocessed groups of straight lines within the boundary of the semantic object image to obtain intersections between the straight lines, and use the pixel coordinates of the intersections as the pixel coordinates of the feature points of the semantic object; wherein the preprocessing includes: 1) selecting a straight line in any group as a baseline, deleting other straight lines whose distance from the baseline is less than a distance threshold, 2) if the number of deleted straight lines is 0, return to 1), otherwise end.

[0073] That is, if the detected object is of three-dimensional type, the line feature selection technology is used to determine the position of the feature points of the semantic object in the pixel coordinate system. Figure 6 As shown, a line detection algorithm first identifies all lines in the image. The slope of each line is calculated, and lines with equal slopes are grouped together. Within each group, a single line is selected as the baseline. The distance between the baseline and the other lines is calculated. Lines that are too close to the baseline are removed, and the same process is repeated for the remaining lines. Each group of lines is extended across the image and the intersection between them is found. This intersection is the fire extinguisher's feature point, and its pixel coordinates are the pixel coordinates of the feature point.

[0074] As Figures 2-5 As an example of the actual scene, semantic objects and semantic map shown in the figure, a round trip experiment is first conducted. The results are as follows Figure 7 The dotted line along the way is the original VINS-Mono trajectory. Since the visual odometry can only obtain relative positions, the original VINS-Mono coordinates cannot be converted to the established corridor world coordinate system. The corrected coordinates are in the world coordinate system, which is consistent with the corridor coordinate system established above, with the right direction being the positive X-axis direction.

[0075] Due to different coordinate systems, the error of the two methods is compared using the error of the starting position after one round trip as the error standard. During the running process, the last position of the round trip is approximately the same as the starting position.

[0076] The original method records the position after initialization, and the starting coordinates are (0.0737138, 0.0179614), and the last coordinates are (1.39808, -4.9185), with an error of 5.0111 m.

[0077] The starting point coordinates of the method provided by the application are (-1.00967, 0.747392), and the last coordinates are (-1.0993, 0.699748), with an error of 0.1015 m. Compared with the original method, the accuracy is greatly improved.

[0078] The round trip experiment is carried out in the corridor, and the experiment lasts for 12 minutes, and the running distance is about 500 meters. The experimental results are shown in Figure 8 The dashed line in the figure is the original VINS-Mono, and the solid line is the running trajectory after correction.

[0079] As can be seen from Figure 8 , the error will continuously accumulate without correction, and the error cannot be automatically eliminated, but continuous correction can ensure that the system does not diverge.

[0080] Now according to the maximum error of the long-time running position estimation system. Taking the y-axis as the error quantification standard, the y-axis range of the narrow corridor is-1.2m to 1.2m, and the actual walking is within-1m to 1m, but when walking in a ring at both ends of the corridor, it will be less than-1m in the negative direction, but not more than 1m. The maximum y-axis in the positioning data is 1.6279m, so the maximum error of the y-axis is estimated to be about 0.6m. It is considered that the x-axis and the y-axis have the same error distribution, so the maximum total error is about 0.85m.

[0081] The z-axis height change is shown in Figure 9 , the z-axis height is relatively stable, and the handheld camera height is about 1.5m, so the z-axis error is about 0.3m.

[0082] The electronic device provided by the embodiment of the application comprises a computer readable storage medium and a processor.

[0083] The computer readable storage medium is used to store executable instructions.

[0084] The processor is used to read the executable instructions stored in the computer readable storage medium, and execute the method as described in any of the above embodiments.

[0085] The embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores computer instructions, and the computer instructions are used for making a processor execute the method in any one of the above embodiments.

[0086] The embodiment of the present application provides a computer program product, including computer programs or instructions, which are executed by a processor to realize the method in any one of the above embodiments.

[0087] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not used to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A visual odometry positioning correction method based on environmental objects as semantic landmarks, characterized by: include: S1, selecting semantic objects from an actual scene, using the world coordinate system of the actual scene as a global coordinate system, recording the semantic labels and global coordinates of feature points of each semantic object to construct a semantic map; wherein the semantic objects have rectangular plane features, and their feature points are the four vertices of the rectangular plane; the semantic objects include a first semantic object, and the first semantic object is unique in the actual scene; S2, correcting the camera pose according to the semantic map, including: S21, inputting the current actual scene image captured by the camera into a pre-trained semantic segmentation model for semantic segmentation to obtain a first semantic object image, and inputting the image into a pre-trained classification model to obtain a semantic label of the first semantic object; S22, obtaining the global coordinates of the feature point of the first semantic object from the semantic map according to the semantic label, and converting it to the camera coordinate system according to the current camera pose provided by the visual odometry to determine whether it is within the current field of view of the camera. If so, proceed to S23, otherwise return to S22; S23, matching feature points of the first semantic object according to their global coordinates and pixel coordinates, calculating the camera pose using the PnP algorithm based on the matching results, and adding the PnP algorithm as a factor to the factor graph to achieve camera pose correction.

2. The method according to claim 1, wherein The semantic object further includes a second semantic object, and the second semantic object is not unique in the actual scene; Step S2 further includes: S21 , inputting the current actual scene image into a pre-trained semantic segmentation model for semantic segmentation to obtain a second semantic object image; S22`, based on the current global pose of the camera provided by the visual odometry, search the semantic map for the N feature points of the second semantic object closest to the camera, and convert them to the camera coordinate system according to the camera pose to determine whether they are within the current field of view of the camera. If at least one second semantic object is within the current field of view of the camera, enter S23`, otherwise return to S22`; where N>1; S23`, matching the feature points of the second semantic object according to their global coordinates and pixel coordinates, calculating the camera pose using the PnP algorithm based on the matching results, and adding it as a factor to the factor graph to achieve camera pose correction.

3. The method according to claim 1 or 2, wherein: If the geometric attribute of the semantic object is a plane, the pixel coordinates of its feature points are determined by extracting all corner points in the semantic object image, screening out the corner points whose distances to the boundary of the semantic object image are less than a distance threshold, and using the pixel coordinates of the four corner points with the smallest distances that correspond one-to-one to the four vertices of the semantic object image as the pixel coordinates of the feature points of the semantic object; If the geometric attribute of the semantic object is three-dimensional, the pixel coordinates of its feature points are determined as follows: identify all straight lines in the semantic object image, group straight lines with equal slopes into the same group, preprocess each group of straight lines, extend the preprocessed groups of straight lines within the boundary of the semantic object image to obtain intersections between the straight lines, and use the pixel coordinates of the intersections as the pixel coordinates of the feature points of the semantic object; wherein the preprocessing includes: 1) selecting a straight line in any group as a baseline, deleting other straight lines whose distance from the baseline is less than a distance threshold, 2) if the number of deleted straight lines is 0, return to 1), otherwise end.

4. The method according to claim 1, wherein The semantic segmentation model is obtained by training a first convolutional neural network using a first data set; the first data set includes actual scene images collected from multiple perspectives and first semantic object images therein; The classification model is obtained by training a second convolutional neural network using a second data set; the second data set includes the first semantic object image and its semantic label in the first data set.

5. The method according to claim 2, wherein The semantic segmentation model is obtained by training a first convolutional neural network using a first data set; the first data set includes actual scene images collected from multiple perspectives and corresponding first and second semantic object images; The classification model is obtained by training a second convolutional neural network using a second data set; the second data set includes the first semantic object image and its semantic label in the first data set.

6. The method according to claim 5, wherein The first and second convolutional neural networks are both YOLO networks.

7. An electronic device, characterized in that: include: Computer-readable storage medium and processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images