Visual odometer positioning correction method based on environmental object as semantic road sign

Abstract

The invention discloses a visual odometer positioning correction method based on an environmental object as a semantic road sign, and belongs to the field of visual localization, and the method comprises the steps: constructing a semantic map through the semantic object in an actual scene, and training a deep learning model for semantic segmentation and classification; semantic segmentation detection and classification detection of a first semantic object are performed on an actual scene image acquired during operation of a positioning system, the first semantic object is used as a semantic road sign to retrieve a matching item from a semantic map, and when the matching item is in a camera view range, pixel coordinates of feature points of the matching item are utilized to perform positioning correction of a visual odometer; according to the method, positioning correction is carried out on the visual odometer based on matching of a semantic segmentation model, a classification model and a map, the application limitation of the visual odometer in an indoor complex environment can be got rid of by using the anti-interference performance of semantic features in the environment, and the robustness and autonomy of a positioning system in the complex environment are improved.

Description

Technical Field

[0001] The present invention belongs to the field of visual positioning, and more specifically, relates to a visual odometer positioning correction method based on environmental objects as semantic landmarks. Background Art

[0002] Autonomous positioning is a core technical issue in the fields of robot navigation and unmanned driving. Autonomous positioning generally refers to the technology of estimating and positioning the position and attitude through internal sensors and algorithms without relying on external signals or information sources. Unlike traditional positioning methods that rely on satellite navigation (such as GPS), autonomous positioning technology can operate in an environment without external positioning signals. The odometer is a module that estimates the device's position and attitude information from sensor data. Its accuracy and robustness play a decisive role in the performance of the autonomous positioning system.

[0003] Among traditional odometers, wheel odometers measure the rotation of wheels to estimate the displacement of vehicles or robots, which can easily cause drift due to wheel slippage; accurate lidar sensors are expensive; inertial sensors estimate position and attitude by integrating acceleration and angular velocity, and over time, measurement errors will gradually accumulate and drift. Visual odometers obtain environmental information through cameras and use changes in image feature points to estimate the motion trajectory of devices or robots in space. They are low-cost and can achieve high-precision robust positioning in complex environments. However, in indoor low-texture environments, the visual odometer calculation method uses the underlying brightness relationship to perform feature description matching, has insufficient anti-interference ability, and is easily affected by factors such as changes in illumination, changes in perspective, and dynamic objects, resulting in feature point matching errors, which will produce cumulative errors in subsequent pose estimation, thus affecting the final positioning results. Summary of the invention

[0004] In order to solve the problem of decreased positioning performance of visual odometer in complex indoor environments due to factors such as illumination changes, feature scarcity and limited field of view, the present invention proposes a visual positioning correction method based on environmental objects as semantic landmarks, aiming to correct the positioning results of the visual odometer through semantic detection network and map matching, get rid of the limitations of the visual odometer in indoor environments, and ultimately improve the robustness and autonomy of the positioning system in complex environments.

[0005] To achieve the above object, according to a first aspect of the present invention, a visual odometer positioning correction method based on environmental objects as semantic landmarks is provided, comprising:

[0006] S1, selecting semantic objects from an actual scene, taking the world coordinate system of the actual scene as a global coordinate system, recording the semantic labels of each semantic object and the global coordinates of the feature points, so as to construct a semantic map; wherein the semantic object has a rectangular plane feature, and its feature points are the four vertices of the rectangular plane; the semantic object includes a first semantic object, and the first semantic object is unique in the actual scene;

[0007] S2, correcting the camera posture according to the semantic map, including:

[0008] S21, inputting the current actual scene image captured by the camera into a pre-trained semantic segmentation model for semantic segmentation to obtain a first semantic object image, and inputting the image into a pre-trained classification model to obtain a semantic label of the first semantic object;

[0009] S22, obtaining the global coordinates of the feature points of the first semantic object from the semantic map according to the semantic label, and converting them to the camera coordinate system according to the current position of the camera provided by the visual odometer to determine whether it is within the current field of view of the camera, if so, entering S23, otherwise returning to S22;

[0010] S23, matching feature points of the first semantic object according to their global coordinates and pixel coordinates, calculating the camera's posture using a PnP algorithm according to the matching results, and adding it as a factor to a factor graph to achieve camera posture correction.

[0011] According to a second aspect of the present invention, there is provided an electronic device, comprising: a computer-readable storage medium and a processor;

[0012] The computer-readable storage medium is used to store executable instructions;

[0013] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method as described in the first aspect.

[0014] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a processor to execute the method as described in the first aspect.

[0015] According to a fourth aspect of the present invention, there is provided a computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the method according to the first aspect is implemented.

[0016] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0017] The method provided by the present invention builds a semantic map through semantic objects in an actual scene, and trains a deep learning model for semantic segmentation and classification; performs semantic segmentation detection and classification detection on a first semantic object on an actual scene image collected during the operation of a positioning system, and uses the first semantic object as a semantic landmark to retrieve a matching item from the semantic map, and when the matching item is within the camera field of view, uses the pixel coordinates of its feature point to perform positioning correction of a visual odometer; the method performs positioning correction on a visual odometer based on matching of a semantic segmentation model, a classification model, and a map, and uses the anti-interference property of semantic features in the environment to get rid of the application limitation of a visual odometer in a complex indoor environment, and improves the robustness and autonomy of a positioning system in a complex environment.

[0018] As a further preferred embodiment of the present invention, the method provided by the present invention also performs semantic segmentation detection of second semantic objects on actual scene images collected during the operation of the positioning system, and retrieves multiple second semantic objects closest to the camera as matching items from the semantic map according to the current posture provided by the visual odometry, and when the matching items are within the field of view of the camera, the pixel coordinates of their feature points are used to perform positioning correction of the visual odometry, thereby further improving the correction accuracy of the camera posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flow chart of a visual positioning correction method based on environmental objects as semantic landmarks provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of an actual scene and a first semantic object therein provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of an actual scene and a second semantic object therein provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of a semantic map provided by an embodiment of the present invention;

[0023] Figure 5 A unique semantic object feature point extraction result diagram provided by an embodiment of the present invention;

[0024] Figure 6 A result diagram of repeated semantic object feature point extraction provided by an embodiment of the present invention;

[0025] Figure 7 A comparison chart of the results of a single round-trip correction experiment and an uncorrected experiment using the method provided by an embodiment of the present invention;

[0026] Figure 8 A comparison chart of the results of multiple round-trip correction experiments and uncorrected experiments using the method provided by the embodiment of the present invention;

[0027] Fig. 9 This is a z-axis result diagram in a global coordinate system of multiple round-trip correction experiments using the method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0029] The embodiment of the present invention provides a visual odometer positioning correction method based on environmental objects as semantic landmarks, such as Figure 1 As shown, including:

[0030] S1, select semantic objects from the actual scene and use them as semantic landmarks, use the world coordinate system of the actual scene as the global coordinate system, record the semantic labels of each semantic object and the global coordinates of the feature points of each semantic object, so as to construct a semantic map;

[0031] The semantic object has a rectangular plane feature, and its feature points are four vertices of the rectangular plane; the semantic object includes a first semantic object, and the first semantic object is unique in the actual scene.

[0032] Specifically, a world coordinate system of the actual scene is established and used as the global coordinate system. Objects with semantic value (i.e., semantic objects) are selected from the actual scene and used as semantic landmarks. Four feature points are selected for each semantic object, and the global coordinates of the feature points of each semantic object and their corresponding semantic labels are recorded to obtain a semantic map.

[0033] To realize the initialization of the positioning system, the semantic objects must include the first semantic object, which is unique in the actual scene. Uniqueness means that there is only one first semantic object in a given environment, such as Figure 2 The house number of the room shown. The characteristic points of the house number are the four vertices of the house number.

[0034] In order to increase the correction accuracy of the camera pose, as a further preferred embodiment of the present invention, the semantic object may further include a second semantic object, and the second semantic object is not unique in the actual scene, that is, the second semantic object may appear repeatedly in the same scene. Figure 2 Fire extinguisher boxes shown repeated at regular intervals.

[0035] In addition, semantic objects should have planar features, such as Figure 2-3The house number and fire extinguisher shown both have plane features; accordingly, the feature points of the semantic object are the four vertices of its rectangular plane.

[0036] The semantic map and world coordinate diagram established according to the actual scene are as follows: Figure 4 As shown, it includes a first semantic object - house number, and a second semantic object - fire extinguisher box. For the convenience of illustration, a triangle is used to represent the fire extinguisher box.

[0037] S2, correcting the camera posture according to the semantic map, including:

[0038] S21, inputting the actual scene image collected in real time by the camera during the operation of the autonomous positioning system into a pre-trained semantic segmentation model for semantic segmentation to obtain a first semantic object image segmented from the current actual scene image, and inputting the first semantic object image into a pre-trained classification model to obtain a semantic label of the first semantic object;

[0039] S22, obtaining the global coordinates of the feature points of the first semantic object from the semantic map according to the semantic label, and converting the feature points of the first semantic object from the global coordinate system to the camera coordinate system according to the current position of the camera to determine whether they are within the current field of view of the camera, if so, entering S23, otherwise returning to S22;

[0040] S23, matching the feature points of the first semantic object according to their global coordinates and pixel coordinates (i.e., coordinates in the pixel coordinate system of the current actual scene image), calculating the camera's posture using the PnP algorithm based on the matching results, and adding it as a factor to the factor graph to achieve camera posture correction.

[0041] When the semantic object further includes a second semantic object, step S2 further includes:

[0042] S21`, inputting the current actual scene image into a pre-trained semantic segmentation model for semantic segmentation to obtain a second semantic object image;

[0043] S22`, searching for feature points of the N second semantic objects closest to the camera in the semantic map according to the current global position of the camera, and converting the feature points of each second semantic object to the camera coordinate system according to the position of the camera to determine whether each second semantic object is within the current field of view of the camera. If at least one second semantic object is within the current field of view of the camera, enter S23`, otherwise return to S22`;

[0044] S23`, matching feature points of the second semantic object according to their global coordinates and pixel coordinates, calculating the camera's posture using the PnP algorithm according to the matching results, and adding it as a factor to the factor graph to achieve camera posture correction.

[0045] The training process of the classification model of the semantic segmentation model is as follows:

[0046] The semantic segmentation model is obtained by training a first convolutional neural network using a first data set; the first data set includes actual scene images collected from multiple perspectives and first semantic object images therein;

[0047] It can be understood that in order to realize the segmentation of the second semantic object image by the semantic segmentation model in S21`, when the semantic object also includes the second semantic object, the training data set of the semantic segmentation model also includes the second semantic object image corresponding to the actual scene image collected from multiple perspectives.

[0048] The classification model is obtained by training a second convolutional neural network using a second data set; the second data set includes the first semantic object image and its semantic label in the first data set.

[0049] That is, scene image data with selected semantic objects is captured from multiple perspectives, and the dataset is used to train semantic segmentation models and classification models:

[0050] First, collect and prepare the image data required for training to train the semantic segmentation model. Use the trained segmentation model to output the segmentation results, save the segmented areas and archive them to obtain the segmented area images; then, use the archived segmentation results to train the classification model. It is worth noting that only unique objects participate in the training process of the classification model.

[0051] The first and second convolutional neural networks may adopt any convolutional neural network. Considering the requirements of calculation accuracy and lightweight, the first and second convolutional neural networks preferably adopt YOLO network.

[0052] Taking the example that both the first and second convolutional neural networks are YOLO networks, a video of an actual scene is recorded and the actual scene pictures that can be used are screened out, and then the house numbers (i.e., the first semantic objects) are annotated using annotation software. After the data set is made, the first YOLO network is trained to obtain a semantic segmentation network to detect the house number image. After the house number image is detected, the corresponding house number needs to be obtained. Therefore, the house number image data set (i.e., the first semantic object image) and its corresponding house number (i.e., the semantic label of the first semantic object image) are used to train the second YOLO network to obtain a classification model, that is, first cut out the house number image detected by the semantic segmentation network, and then use the house numbers of different house number images as different categories to train the second YOLO network to obtain a classification model.

[0053] When the semantic object also includes a fire extinguisher (ie, the second semantic object), the training data set of the semantic segmentation model also includes fire extinguisher images (ie, the second semantic object images) corresponding to actual scene images collected from multiple perspectives.

[0054] In step S2, during the operation of the visual odometry, the scene image is collected in real time by the camera and segmentation detection and classification detection of semantic objects are performed on it. The global posture information provided by the visual odometry is used to retrieve matching items from the semantic map. When the match is successful, the feature points are matched according to the pixel coordinates of the feature points of the semantic object and the global coordinates to perform camera posture correction, thereby realizing the positioning correction of the visual odometry.

[0055] It is worth noting that the above process implements differentiated processing according to the semantic object category:

[0056] (1) If the semantic object detected by semantic segmentation is unique in the actual scene, that is, the detected semantic object is the first semantic object, then according to the current global posture and the output result of the classification model, find the matching item of the first semantic object in the semantic map:

[0057] According to the semantic label of the first semantic object, the global coordinates of the feature points of the first semantic object are searched in the semantic map. According to the current position of the camera and the imaging principle of the camera, the global coordinates of the feature points of the first semantic object are converted to the camera coordinate system to determine whether the first semantic object is within the field of view of the camera.

[0058] That is, at this time, the first semantic object is the semantic landmark. The global coordinates of the feature points of the semantic landmark are obtained and converted to the camera coordinates. If the coordinates x of any feature point are c ,y cOr if zc exceeds the current field of view of the camera, it is considered that the first semantic object currently observed by the camera is not the object selected in the semantic map, that is, the output result of the classification model does not match the actual first semantic object and cannot be used for visual positioning correction, then the above-mentioned actual scene image acquisition, segmentation and detection processing steps are returned; otherwise, it can be used for visual positioning correction: the pixel coordinates of the feature points of the first semantic object are used to match the feature points with the global coordinates, and the pixel coordinates of the feature points are matched with the global coordinates. The PnP algorithm is used to calculate the camera's pose, and the position of the camera calculated by the PnP algorithm is used as a factor, and the factor is added to the factor graph to achieve pose correction.

[0059] When matching feature points, a unified sorting strategy is used, that is, the extracted feature points under the pixel coordinates and the saved feature point coordinates are sorted in a clockwise direction, with the pixel point in the upper left corner as the first reference point. In this way, the pixel coordinates of the feature points are accurately matched with the global coordinates.

[0060] The camera imaging principle is:

[0061]

[0062] Among them, x w ,y w ,z w are the coordinates of any point in the global coordinate system, R, t are the rotation and translation of the current position of the camera, dx and dy represent the physical size of each pixel on the horizontal axis x and the vertical axis y respectively, f is the focal length, u0, v0 are the principal point coordinates, and u, v are the pixel coordinates.

[0063] The calculation formula from global coordinates to camera coordinates is:

[0064]

[0065] Among them, x c ,y c ,z c is the coordinate in the camera coordinate system.

[0066] (2) If the semantic object detected by semantic segmentation is not unique in the actual scene, that is, the detected semantic object is a second semantic object, then according to the current global pose and the output result of the semantic segmentation model, find the matching item of the second semantic object in the semantic map:

[0067] At this time, there is no output information of the classification network, only the output information of the segmentation network, that is, the image of the second semantic object. Therefore, according to the current global pose provided by the visual odometer, the N (N>1) second semantic objects closest to the camera are selected in the semantic map, and the global coordinates of the feature points of each second semantic object are converted to the camera coordinate system (the conversion-related calculations are the same as above), and it is determined one by one whether they are within the camera's field of view. If at least one second semantic object is within the camera's field of view, it is considered that the second semantic object currently observed by the camera is the object selected in the semantic map, which can be used for visual positioning correction: the pixel coordinates of the feature points of the second semantic object are used to match the feature points with the global coordinates, and the matching results of the pixel coordinates of the feature points with the global coordinates are used to calculate the camera's pose using the PnP algorithm, and the position of the camera calculated by the PnP algorithm is used as a factor, and the factor is added to the factor graph to achieve pose correction; otherwise, it is considered that the second semantic object currently observed by the camera is not the object selected in the semantic map, that is, the output result of the classification model does not match the actual second semantic object and cannot be used for visual positioning correction, and then the above-mentioned actual scene image acquisition and segmentation processing steps are returned. That is, when the semantic object detected by the semantic segmentation also includes a second semantic object, and the second semantic object currently observed by the camera is an object selected in the semantic map, the second semantic object is used to perform posture correction again, thereby further improving the correction accuracy.

[0068] For example, select the five fire extinguisher boxes closest to the current posture in the semantic map, and determine whether they are within the field of view one by one. If at least one fire extinguisher box is within the camera's field of view, the pixel coordinates of its feature points can be used for visual positioning correction.

[0069] The pixel coordinates of the feature points of the first and second semantic objects can be calculated using existing methods. Considering that the semantic objects have rectangular plane features, the geometric attributes of the semantic objects may be plane or three-dimensional. To improve the calculation accuracy, as a further preferred solution of the present invention:

[0070] If the geometric attribute of the semantic object is a plane, the pixel coordinates of its feature points are determined by extracting all corner points in the semantic object image, screening out the corner points whose distance to the boundary of the semantic object image is less than a distance threshold, and taking the pixel coordinates of the four corner points with the shortest distance that correspond one-to-one to the four vertices of the semantic object image as the pixel coordinates of the feature points of the semantic object.

[0071] That is, if the detected object is of the plane type, the point feature selection technique is used to determine the pixel coordinates of the feature points of the semantic object. Figure 5As shown, the rectangular bounding box of the house number is extracted from the output of the segmentation model, the image is converted into grayscale mode, and the area within the bounding box is cropped to obtain the house number image. Subsequently, the corner detection algorithm is applied to extract the corner points from the cropped image. The corner points whose distance from the boundary of the house number image is less than the distance threshold are preliminarily screened out. The distance threshold is S / K+b, where S is the area of ​​the region, K is the proportional coefficient, and b is the bias coefficient, which can all be set according to experience. That is, the larger the area of ​​the region, the larger the distance threshold. Finally, based on the distance between the feature point and the vertices of the rectangle (i.e., the vertices of the bounding box), the four corner points with the smallest distance are screened, which are the four feature points of the house number, and their pixel coordinates are the pixel coordinates of the four feature points of the house number.

[0072] If the geometric attribute of the semantic object is three-dimensional, the pixel coordinates of its feature points are determined as follows: identify all straight lines in the semantic object image, group straight lines with equal slopes into the same group, preprocess each group of straight lines, extend the preprocessed groups of straight lines within the boundary of the semantic object image to obtain intersections between the straight lines, and use the pixel coordinates of the intersections as the pixel coordinates of the feature points of the semantic object; wherein the preprocessing includes: 1) selecting a straight line in any group as a baseline, deleting other straight lines whose distance from the baseline is less than a distance threshold, 2) if the number of deleted straight lines is 0, return to 1), otherwise end.

[0073] That is, if the detected object is of a three-dimensional type, the line feature selection technique is used to determine the position of the feature points of the semantic object in the pixel coordinate system. Figure 6 As shown in the figure, first, all the lines in the image are identified by the line detection algorithm, and the slope of each line is calculated, and the lines with equal slopes are grouped into the same group. In each group, a line is selected as the baseline, and the distance between the baseline and other lines is calculated. After removing other lines that are too close to the baseline, the remaining lines are processed in the same way. By extending each group of lines in the image range and finding the intersection between each group of lines, the intersection is the feature point of the fire extinguisher, and its pixel coordinates are the pixel coordinates of the feature point.

[0074] As Figure 2-5 As an example, we first conduct a round trip experiment. The results are shown in Figure 7 As shown in the figure. The dotted track on the way is the original VINS-Mono. Since the visual odometer can only obtain relative positions, the coordinates of the original VINS-Mono method cannot be transformed into the established corridor world coordinate system. The corrected coordinates are in the world coordinate system, which is consistent with the corridor coordinate system established in the previous article. The right is the positive direction of the X axis.

[0075] Due to the different coordinate systems, the error between the two methods is compared using the error between the position after one round trip and the starting position as the error standard. During the operation, the final position of the round trip is roughly the same as the starting position.

[0076] After the original method is initialized, the position is recorded again. The starting coordinates are (0.0737138, 0.0179614), and the final coordinates are (1.39808, -4.9185), with an error of 5.0111m.

[0077] The starting point coordinates of the method provided by the present invention are (-1.00967, 0.747392), and the final coordinates are (-1.0993, 0.699748), with an error of 0.1015m. Compared with the original method, the accuracy is greatly improved.

[0078] A round-trip experiment was conducted in the corridor. The experiment lasted 12 minutes and the running distance was about 500 meters. The experimental results are as follows: Figure 8 The dotted line in the figure is the original VINS-Mono, and the solid line is the corrected running trajectory.

[0079] from Figure 8 It can be seen that when not corrected, the errors will continue to accumulate and cannot be eliminated automatically, while continuous correction can ensure that the system does not diverge.

[0080] Now, the maximum error of the system is estimated based on the long-term running position. Taking the y-axis as the error quantification standard, the y-axis range of the narrow corridor is -1.2m to 1.2m, and the actual walking range is between -1m and 1m, but when walking in a circle at both ends of the corridor, it will be less than -1m in the negative direction, but not more than 1m. In the positioning data, the maximum positive direction of the y-axis is 1.6279m, so the maximum error of the y-axis operation is estimated to be about 0.6m. It is assumed that the x-axis and y-axis have the same error distribution, so the maximum total error is about 0.85m.

[0081] The z-axis height changes as Fig. 9 As shown, the z-axis height is relatively stable, and the height of the handheld camera is about 1.5m, so the z-axis error is about 0.3m.

[0082] An embodiment of the present invention provides an electronic device, including: a computer-readable storage medium and a processor;

[0083] The computer-readable storage medium is used to store executable instructions;

[0084] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method described in any of the above embodiments.

[0085] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method described in any of the above embodiments.

[0086] An embodiment of the present invention provides a computer program product, including a computer program or instructions, and when the computer program or instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0087] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual odometer positioning correction method based on environmental objects as semantic landmarks, characterized in that: include: S1, selecting semantic objects from an actual scene, taking the world coordinate system of the actual scene as a global coordinate system, recording the semantic labels of each semantic object and the global coordinates of the feature points, so as to construct a semantic map; wherein the semantic object has a rectangular plane feature, and its feature points are the four vertices of the rectangular plane; the semantic object includes a first semantic object, and the first semantic object is unique in the actual scene; S2, correcting the camera posture according to the semantic map, including: S21, inputting the current actual scene image captured by the camera into a pre-trained semantic segmentation model for semantic segmentation to obtain a first semantic object image, and inputting the image into a pre-trained classification model to obtain a semantic label of the first semantic object; S22, obtaining the global coordinates of the feature points of the first semantic object from the semantic map according to the semantic label, and converting them to the camera coordinate system according to the current position of the camera provided by the visual odometer to determine whether it is within the current field of view of the camera, if so, entering S23, otherwise returning to S22; S23, matching feature points of the first semantic object according to their global coordinates and pixel coordinates, calculating the camera's posture using a PnP algorithm according to the matching results, and adding it as a factor to a factor graph to achieve camera posture correction.

2. The method according to claim 1, characterized in that The semantic object further includes a second semantic object, and the second semantic object is not unique in the actual scene; Step S2 also includes: S21`, inputting the current actual scene image into a pre-trained semantic segmentation model for semantic segmentation to obtain a second semantic object image; S22`, searching for the feature points of the N second semantic objects closest to the camera in the semantic map according to the current global position of the camera provided by the visual odometer, and converting them to the camera coordinate system according to the position of the camera to determine whether they are within the current field of view of the camera. If at least one second semantic object is within the current field of view of the camera, enter S23`, otherwise return to S22`; wherein N>1; S23`, matching feature points of the second semantic object according to their global coordinates and pixel coordinates, calculating the camera's posture using the PnP algorithm according to the matching results, and adding it as a factor to the factor graph to achieve camera posture correction.

3. The method according to claim 1 or 2, characterized in that If the geometric attribute of the semantic object is a plane, the pixel coordinates of its feature points are determined by extracting all corner points in the semantic object image, screening out the corner points whose distances to the boundary of the semantic object image are less than a distance threshold, and taking the pixel coordinates of the four corner points with the smallest distances that correspond one-to-one to the four vertices of the semantic object image as the pixel coordinates of the feature points of the semantic object; If the geometric attribute of the semantic object is three-dimensional, the pixel coordinates of its feature points are determined as follows: identify all straight lines in the semantic object image, group straight lines with equal slopes into the same group, preprocess each group of straight lines, extend the preprocessed groups of straight lines within the boundary of the semantic object image to obtain intersections between the straight lines, and use the pixel coordinates of the intersections as the pixel coordinates of the feature points of the semantic object; wherein the preprocessing includes: 1) selecting a straight line in any group as a baseline, deleting other straight lines whose distance from the baseline is less than a distance threshold, 2) if the number of deleted straight lines is 0, return to 1), otherwise end.

4. The method according to claim 1, characterized in that The semantic segmentation model is obtained by training a first convolutional neural network using a first data set; the first data set includes actual scene images collected from multiple perspectives and first semantic object images therein; The classification model is obtained by training a second convolutional neural network using a second data set; the second data set includes the first semantic object image and its semantic label in the first data set.

5. The method according to claim 2, characterized in that The semantic segmentation model is obtained by training a first convolutional neural network using a first data set; the first data set includes actual scene images collected from multiple perspectives and corresponding first and second semantic object images; The classification model is obtained by training a second convolutional neural network using a second data set; the second data set includes the first semantic object image and its semantic label in the first data set.

6. The method according to claim 5, characterized in that The first and second convolutional neural networks are both YOLO networks.

7. An electronic device, characterized in that: include: A computer readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute the method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images