Camera-based real-time semantic segmentation and geolocation method and system for street scene features

Abstract

The application belongs to the technical field of computer vision, and particularly relates to a camera-based real-time semantic segmentation and geographic positioning method and system for street scene objects, which comprises the following steps: S1, image acquisition and preprocessing; S2, semantic segmentation; S3, panoramic monocular depth estimation and scale recovery; S4, target detection and pixel extraction; S5, spatial coordinate solution; and S6, Kalman filtering fusion based on time sequence, wherein the finally converged coordinates of the output filter are taken as the final geographic position of the object. Different from vehicle positioning technology which depends on prior high-precision map matching, the application uses panoramic depth estimation and spatial coordinate solution algorithm, can directly deduce the three-dimensional spatial position of the object from two-dimensional image pixels without prior map data, and combines GNSS / IMU data to calculate the WGS84 absolute geographic coordinates of the object in real time.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, specifically relating to a camera-based method and system for real-time semantic segmentation and geolocation of street scene features. Background Technology

[0002] With the rapid development of smart city construction and autonomous driving technology, high-definition maps (HDMap) and urban digital twins place extremely high demands on the surveying and updating of road infrastructure (such as streetlights, traffic signs, manhole covers, and trash cans). The ability to quickly, cost-effectively, and accurately acquire the categories and absolute geographical locations (GIS coordinates) of these features is a key requirement in the current surveying and geographic information field. Existing visual positioning technologies primarily focus on the vehicle's own positioning. These methods typically assume the existence of a high-definition map and correct the vehicle's position by matching features in the image (such as lane lines) with the prior map. However, in scenarios where maps are missing, new roads are built, or features have changed, these methods relying on prior data fail to function, and they cannot proactively discover and calculate the coordinates of unknown features. Summary of the Invention

[0003] The main objective of this invention is to provide a camera-based real-time semantic segmentation and geolocation method and system for street scene features, aiming to solve the technical problems existing in current mobile measurement technologies, such as the high cost of LiDAR equipment, the limited field of view of ordinary monocular cameras, and the inability of traditional visual algorithms to effectively handle boundary fragmentation and depth loss in panoramic images. The technical solution is as follows:

[0004] A camera-based real-time semantic segmentation and geolocation method for street scene features includes the following steps:

[0005] Step S1: Image acquisition and preprocessing;

[0006] Step S2: Perform semantic segmentation;

[0007] Step S3: Panoramic monocular depth estimation and scale recovery;

[0008] Step S4: Target detection and pixel extraction;

[0009] Step S5: Solve spatial coordinates;

[0010] Step S6: Based on time-series Kalman filter fusion, the final converged coordinates are output as the final geographical location of the feature.

[0011] Preferably, step S1, image acquisition and preprocessing, includes:

[0012] S101. Use an onboard multi-lens panoramic camera to collect video stream data around the vehicle, and at the same time obtain the vehicle's real-time latitude and longitude coordinates and three-axis attitude through a GNSS / IMU integrated navigation system; and use a hardware second pulse signal to trigger camera exposure to ensure that the exposure time of each frame of panoramic image is strictly aligned with the timestamp of the GNSS / IMU data.

[0013] S102. The original images acquired from multiple lenses are stitched together and distortion corrected using a pre-calculated lookup table to generate a panoramic image with equidistant cylindrical projection.

[0014] ;

[0015] (u,v): pixel coordinates;

[0016] : Output panoramic image pixel values;

[0017] : The image of the k-th shot;

[0018] , A pre-computed coordinate mapping lookup table maps panoramic pixels back to original lens pixels.

[0019] Preferably, step S2, semantic segmentation and feature adaptation, includes:

[0020] S201. The output panoramic image is input into the SegFormer-B2 semantic segmentation network based on the Transformer architecture. A cyclic padding mechanism is introduced in the network's convolutional layers and Transformer modules: that is, when processing the leftmost pixel of the image, the rightmost pixel of the image is used for padding convolution, and vice versa.

[0021] The expression for filling the loop is as follows:

[0022] ;

[0023] : The filled feature tensor;

[0024] W: Width of the panoramic image;

[0025] Mod: Modulo operation, which enables logical connectivity between the left and right boundaries;

[0026] S202. The model was pre-trained on the Cityscapes dataset and fine-tuned using data augmentation algorithms targeting panoramic distortion (small stretching at the equator, large stretching at the poles);

[0027] S203. Output: Output a semantic mask image with the same resolution as the original image. , where each pixel value Corresponding to a specific category ID;

[0028] The semantic mask diagram is represented as follows:

[0029] ;

[0030] Network: refers to a deep neural network used to perform semantic segmentation tasks;

[0031] ArgMax: In mathematics, it refers to the index corresponding to the maximum value.

[0032] Preferably, in step S3, the panoramic image is input into the DINOV3 panoramic depth estimation network. This network adopts a dual-branch structure, which processes the ERP image and cube map projection separately to generate a single-channel depth map. , where pixels The grayscale value directly corresponds to the straight-line distance of this point from the optical center of the camera. ;

[0033] Geometric Scale Recovery:

[0034] ;

[0035] S: Absolute scaling factor;

[0036] The mounting height of the camera's optical center relative to the ground;

[0037] The pitch angle corresponding to the ground point;

[0038] The relative depth value output by the deep network, i.e., the dimensionless value;

[0039] Absolute depth map generation:

[0040] ;

[0041] : pixel The corresponding real-world straight-line distance.

[0042] Preferably, step S4, target detection and pixel extraction, comprises the following steps:

[0043] S41. Perform connected component analysis on the mask graph output in step S2;

[0044] S42. Extract all independent connected regions belonging to the target category;

[0045] S43. Calculate the pixel coordinates of the geometric centroid or bottom center point of each connected region. :

[0046] ;

[0047] The set of pixels representing the k-th target in the panoramic image;

[0048] The row coordinates of the bottom of the target;

[0049] : Horizontal center column coordinates at the bottom of the target;

[0050] S44. From the depth map Read the distance value corresponding to the coordinates:

[0051] ;

[0052] The straight-line distance from the root of the target to the optical center of the camera.

[0053] Preferably, step S5 uses pixel coordinates, depth values, and camera pose to calculate the absolute coordinates of the target. The calculation process is as follows:

[0054] Converting pixel coordinates to spherical coordinates:

[0055] ;

[0056] W, H: Width and height of the panoramic image;

[0057] azimuth, range ;

[0058] Pitch angle, range ;

[0059] Spherical coordinates to camera coordinate system:

[0060] ;

[0061] : Three-dimensional coordinates in the camera system;

[0062] Axial displacement: Camera system to vehicle engine system:

[0063] Correcting the difference in axial definitions between the camera system and the vehicle system (IMU):

[0064] ;

[0065] : Three-dimensional coordinates in a machine system;

[0066] definition: To the right of the camera's 3D coordinates For the front of the three-dimensional coordinates, The sky is in three-dimensional coordinates;

[0067] Transformation logic: The "front" of the camera becomes the "front" of the car, and the "top" of the camera becomes the "top" of the car;

[0068] Attitude compensation:

[0069] Constructing a rotation matrix using IMU attitude angles :

[0070] ;

[0071] The target is offset eastward relative to the center of the vehicle.

[0072] The target is offset northward relative to the center of the vehicle.

[0073] The target is offset upwards relative to the center of the vehicle;

[0074] The vehicle's current heading, pitch, and roll angles;

[0075] Single-frame absolute coordinate calculation:

[0076] Calculate the single-frame observation value of the target by combining the vehicle's GNSS coordinates:

[0077] ;

[0078] This refers to the Earth's average radius;

[0079] : Target latitude and longitude measured in a single frame;

[0080] : The vehicle's current latitude and longitude.

[0081] Preferably, step S6 is based on time-series Kalman filter fusion, and the steps are as follows:

[0082] S601. Using the shift algorithm, the object identified in the current frame is matched with the object in the historical frame tracking queue based on the proximity of positions;

[0083] S602. Calculate the coordinates for each frame. As an input filter for the observed values;

[0084] s603. Set the observation covariance matrix The closer the observation is to the camera, the higher its weight.

[0085] S604. When the target moves out of the camera's field of view, the coordinates at which the output filter finally converges are taken as the final geographical location of the feature.

[0086] Preferably, the state vector With observation vector :

[0087] ;

[0088] : Optimal position estimate after filtering;

[0089] Virtual drift velocity, used to absorb systematic error trends;

[0090] Range-adaptive observation noise :

[0091] ;

[0092] ;

[0093] Standard deviation of reference measurement noise;

[0094] Distance attenuation coefficient;

[0095] Horizontal distance;

[0096] Status Update:

[0097] ;

[0098] Kalman gain, with distance Increase Enlargement, leading to Decrease;

[0099] Final result extraction:

[0100] ;

[0101] , : The final output coordinates.

[0102] A camera-based real-time semantic segmentation and geolocation system for street scene features includes an image acquisition unit, a positioning and orientation unit, a computation unit, and an output unit.

[0103] Image acquisition unit: Uses a panoramic camera to acquire images;

[0104] Positioning and attitude determination unit: A dual-antenna RTK-GNSS / IMU integrated navigation system is used for positioning and attitude determination measurements;

[0105] Computational Unit: Utilizing panoramic depth estimation and spatial coordinate calculation algorithms, it directly infers the three-dimensional spatial location of ground features from two-dimensional image pixels without prior map data, and calculates the WGS84 absolute geographic coordinates of ground features in real time by combining GNSS / IMU data; it also utilizes the Kalman filtering algorithm to take advantage of the multi-view observation brought by vehicle movement, automatically smooths single-frame observation noise, and outputs high-confidence geographic information data.

[0106] Output unit: Visualizes the results.

[0107] Preferably, the image acquisition unit adopts a hard-triggered mode, where the GNSS receiver sends q PPS signals per second to the camera trigger interface to force the camera to expose, ensuring that the center exposure time of each panoramic image has an error of less than a set value compared to the timestamp of the GNSS / IMU data;

[0108] Positioning and attitude determination unit: The IMU output frequency is set to 100Hz to provide the vehicle's real-time roll angle, pitch angle and yaw angle;

[0109] Computing unit: Employs NVIDIA Jetson AGX Orin embedded computing module.

[0110] Compared with the prior art, the beneficial effects of this application are as follows:

[0111] (1) Achieve low-cost, comprehensive land cover survey:

[0112] By using multi-lens panoramic cameras to replace expensive LiDAR, hardware costs can be significantly reduced while achieving comprehensive monitoring of the vehicle's surroundings. The comprehensive environmental coverage solves the problem that ordinary vehicle cameras have difficulty capturing side and rear views as well as high-altitude facilities (such as cantilevered road signs and traffic lights).

[0113] (2) Solving the problems of "seams" and distortion in panoramic images:

[0114] To address the topological characteristic of continuous panoramic images (equidistant cylindrical projection), a semantic segmentation strategy based on circular padding is proposed to eliminate image continuation. The fragmentation effect at the boundary ensures the complete identification of cross-boundary features, overcoming the problem of low edge recognition rate of traditional algorithms in panoramic images.

[0115] (3) Achieve graph-independent monocular absolute localization:

[0116] Unlike vehicle positioning technologies that rely on prior high-precision maps for matching, this invention utilizes panoramic depth estimation and spatial coordinate calculation algorithms to directly infer the three-dimensional spatial location of ground features from two-dimensional image pixels without prior map data, and calculates the WGS84 absolute geographic coordinates of ground features in real time by combining GNSS / IMU data.

[0117] (4) Improve the accuracy and robustness of pure vision positioning

[0118] To address the nonlinear growth of error in visual ranging with increasing distance, a distance-weighted multi-frame fusion mechanism (such as an improved Kalman filter) is proposed. This mechanism leverages the advantages of multi-view observations brought about by vehicle motion to automatically smooth single-frame observation noise and output high-confidence geographic information data. Attached Figure Description

[0119] Figure 1 For flowcharts;

[0120] Figure 2 A geometric diagram illustrating the principle of calculating the geographic location of features based on the pixels of a panoramic image;

[0121] Figure 3 A schematic diagram illustrating the mapping between target and ground feature localization results extracted from panoramic video semantic segmentation. Detailed Implementation

[0122] To better understand the purpose, principles, features, and advantages of this invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The examples given are only for explaining the invention and are not intended to limit the scope of the invention.

[0123] This invention provides a camera-based method and system for real-time semantic segmentation and geolocation of street scene features. The system is mounted on the roof of a mobile survey vehicle (or a regular inspection vehicle) and automatically identifies and calculates the coordinates of road facilities (such as streetlights, manhole covers, and traffic signs) while the vehicle is in motion.

[0124] 1. Hardware environment and system integration:

[0125] To verify the feasibility of this system, the following hardware configuration is used in this embodiment:

[0126] (1) Image acquisition unit: A 5-lens panoramic camera with a total of 24 million pixels and an output frame rate of 30fps is used. The camera is mounted on a roof bracket and is fixed at a height of 2.2 meters from the ground.

[0127] (2) Positioning and attitude determination unit: adopts a dual-antenna RTK-GNSS / IMU integrated navigation system (model example: NovAtelSPAN).

[0128] Positioning accuracy: Horizontal accuracy 1cm+1ppm in RTK mode.

[0129] Attitude update frequency: The IMU output frequency is set to 100Hz to provide the vehicle's real-time roll, pitch, and yaw / heading angles.

[0130] (3) Edge computing unit: The NVIDIA Jetson AGX Orin embedded computing module is used. All system algorithms (stitching, segmentation, and positioning) run locally in real time on this module without needing to be uploaded to the cloud for processing.

[0131] (4) Synchronization triggering mechanism: Hard triggering mode is adopted. The GNSS receiver sends 5 PPS (pulse per second) signals to the camera trigger interface to force the camera to expose, ensuring that the center exposure time of each panoramic image has an error of less than 1ms with the timestamp of the GNSS / IMU data.

[0132] To ensure consistency in the derivation logic, the following definitions are used throughout the text:

[0133] Pixel coordinate system Top left corner of the origin To the right, Downward (unit: pixel).

[0134] Camera coordinate system The origin is the optical center. right, superior, Front (unit: m).

[0135] Vehicle body coordinate system The origin is the center of the car. right, forward, Day (IMU standard, unit: m).

[0136] Navigation coordinate system Northeast Celestial Coordinate System (unit: m).

[0137] Geographic coordinate system WGS84 latitude and longitude (unit: degree).

[0138] A camera-based real-time semantic segmentation and geolocation method for street scene features includes the following steps:

[0139] Step S1: Image acquisition and preprocessing;

[0140] S101. Use an onboard multi-lens panoramic camera to collect video stream data around the vehicle, and at the same time obtain the vehicle's real-time latitude and longitude coordinates and three-axis attitude through a GNSS / IMU integrated navigation system; and use a hardware second pulse signal to trigger camera exposure to ensure that the exposure time of each frame of panoramic image is strictly aligned with the timestamp of the GNSS / IMU data.

[0141] Hardware time synchronization:

[0142] ;

[0143] : The timestamp (UnixTimestamp) of the corresponding sensor.

[0144] Hardware-triggered synchronization error (typically) ), which can be disregarded.

[0145] S102. The original images acquired from multiple lenses are stitched together and distortion corrected using a pre-calculated lookup table to generate a panoramic image with equidistant cylindrical projection.

[0146] ;

[0147] (u,v): pixel coordinates;

[0148] : Output panoramic image pixel values;

[0149] : The image of the k-th shot;

[0150] , A pre-computed coordinate mapping lookup table maps panoramic pixels back to original lens pixels.

[0151] Step S2: Perform semantic segmentation;

[0152] S201. The output panoramic image is input into the SegFormer-B2 semantic segmentation network based on the Transformer architecture. A cyclic padding mechanism is introduced in the network's convolutional layers and Transformer modules: that is, when processing the leftmost pixel of the image, the rightmost pixel of the image is used for padding convolution, and vice versa.

[0153] The expression for filling the loop is as follows:

[0154] ;

[0155] : The filled feature tensor;

[0156] W: Width of the panoramic image;

[0157] ModW: Takes the modulo of the width to achieve logical connectivity between the left and right boundaries;

[0158] S202. The model was pre-trained on the Cityscapes dataset and fine-tuned using data augmentation algorithms targeting panoramic distortion (small stretching at the equator, large stretching at the poles);

[0159] S203. Output: Output a semantic mask image with the same resolution as the original image. , where each pixel value Corresponding to specific category IDs, such as 1=streetlight, 2=traffic sign.

[0160] The semantic mask diagram is represented as follows:

[0161] .

[0162] Step S3: Panoramic monocular depth estimation and scale recovery;

[0163] The panoramic image is input into the DINOV3 panoramic depth estimation network, which employs a dual-branch structure to process the ERP image and cubemap projection separately, generating a single-channel depth map. , where pixels The grayscale value directly corresponds to the straight-line distance of this point from the optical center of the camera. ;

[0164] Geometric Scale Recovery:

[0165] ;

[0166] S: Absolute scaling factor;

[0167] The mounting height of the camera's optical center relative to the ground;

[0168] The pitch angle corresponding to the ground point;

[0169] The relative depth value output by the deep network, i.e., the dimensionless value;

[0170] Absolute depth map generation:

[0171] ;

[0172] : pixel The corresponding real-world straight-line distance.

[0173] Step S4: Target detection and pixel extraction;

[0174] S41. Perform connected component analysis on the mask graph output in step S2;

[0175] S42. Extract all independent connected regions belonging to the target category;

[0176] S43. Calculate the pixel coordinates of the geometric centroid or bottom center point of each connected region. :

[0177] ;

[0178] The set of pixels representing the k-th target in the panoramic image;

[0179] The row coordinates of the bottom of the target;

[0180] : Horizontal center column coordinates at the bottom of the target;

[0181] S44. From the depth map of step S3 Read the distance value corresponding to the coordinates:

[0182] ;

[0183] The straight-line distance from the root of the target to the optical center of the camera.

[0184] Step S5: Solve spatial coordinates;

[0185] The absolute coordinates of the target are calculated using pixel coordinates, depth values, and camera pose. The calculation process is as follows:

[0186] Converting pixel coordinates to spherical coordinates:

[0187] ;

[0188] W, H: Width and height of the panoramic image;

[0189] azimuth, range ;

[0190] Pitch angle, range ;

[0191] Spherical coordinates to camera coordinate system:

[0192] ;

[0193] : Three-dimensional coordinates in the camera system;

[0194] Axial displacement: Camera system to vehicle engine system:

[0195] Correcting the difference in axial definitions between the camera system and the vehicle system (IMU):

[0196] ;

[0197] : Three-dimensional coordinates in a machine system;

[0198] definition: To the right of the camera's 3D coordinates For the front of the three-dimensional coordinates, The sky is in three-dimensional coordinates;

[0199] Transformation logic: The "front" of the camera becomes the "front" of the car, and the "top" of the camera becomes the "top" of the car;

[0200] Attitude compensation:

[0201] Constructing a rotation matrix using IMU attitude angles :

[0202] ;

[0203] The target is offset eastward relative to the center of the vehicle.

[0204] The target is offset northward relative to the center of the vehicle.

[0205] The vehicle's current heading, pitch, and roll angles;

[0206] Single-frame absolute coordinate calculation:

[0207] Calculate the single-frame observation value of the target by combining the vehicle's GNSS coordinates:

[0208] ;

[0209] This refers to the Earth's average radius;

[0210] : Target latitude and longitude measured in a single frame;

[0211] : The vehicle's current latitude and longitude.

[0212] Step S6: Time-based Kalman filter fusion, the final convergence coordinates of the output filter are taken as the final geographical location of the feature.

[0213] The time-based Kalman filter fusion process involves the following steps:

[0214] S601. Using the shift algorithm, the object identified in the current frame is matched with the object in the historical frame tracking queue based on the proximity of positions;

[0215] S602. Calculate the coordinates for each frame. As an input filter for the observed values;

[0216] s603. Set the observation covariance matrix The closer the observation is to the camera, the higher its weight.

[0217] State vector With observation vector :

[0218] ;

[0219] : Optimal position estimate after filtering;

[0220] Virtual drift velocity, used to absorb systematic error trends;

[0221] Range-adaptive observation noise :

[0222] ;

[0223] ;

[0224] Standard deviation of reference measurement noise;

[0225] Distance attenuation coefficient;

[0226] Horizontal distance;

[0227] Status Update:

[0228] ;

[0229] Kalman gain, with distance Increase Enlargement, leading to Decrease;

[0230] Final result extraction:

[0231] ;

[0232] , : The final output coordinates.

[0233] S604. When the target moves out of the camera's field of view, the coordinates at which the output filter finally converges are taken as the final geographical location of the feature.

[0234] A camera-based real-time semantic segmentation and geolocation system for street scene features includes an image acquisition unit, a positioning and orientation unit, a computation unit, and an output unit.

[0235] Image acquisition unit: Uses a panoramic camera to acquire images;

[0236] Positioning and attitude determination unit: A dual-antenna RTK-GNSS / IMU integrated navigation system is used for positioning and attitude determination measurements;

[0237] Computational unit: includes panoramic stitching unit, depth perception unit, coordinate calculation and fusion module;

[0238] The panoramic stitching unit generates equidistant cylindrical projection images in real time; the semantic perception unit runs a neural network with a cyclic filling mechanism to output a land cover category mask; the depth perception unit calculates the relative spatial distances of land cover features. The coordinate calculation and fusion module performs spatial coordinate transformations and weights and fuses multiple frames of observation data based on vehicle movement trajectories to calculate coordinate location information and output structured GIS geographic information data.

[0239] Using panoramic depth estimation and spatial coordinate calculation algorithms, the three-dimensional spatial position of ground features is directly inferred from the pixels of two-dimensional images without prior map data. Combined with GNSS / IMU data, the WGS84 absolute geographic coordinates of ground features are calculated in real time. Using the Kalman filter algorithm, the multi-view observation advantage brought by vehicle movement is taken advantage of to automatically smooth the observation noise of a single frame and output high-confidence geographic information data.

[0240] Output unit: Visualizes the results.

[0241] Image acquisition unit: Employs hard-trigger mode, the GNSS receiver sends 5 PPS signals per second to the camera trigger interface to force camera exposure, ensuring that the center exposure time of each panoramic image has an error of less than 1ms from the timestamp of the GNSS / IMU data;

[0242] Positioning and attitude determination unit: The IMU output frequency is set to 100Hz to provide the vehicle's real-time roll angle, pitch angle and yaw angle;

[0243] Computing unit: Employs NVIDIA Jetson AGX Orin embedded computing module.

[0244] The foregoing has provided a more detailed description of the objectives, technical solutions, and beneficial effects of this invention. It should be understood that the above descriptions are merely specific embodiments of this invention and are intended only to aid in understanding the method and core ideas of this invention. They are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A camera-based method for real-time semantic segmentation and geolocation of street scene features, characterized in that, Includes the following steps: Step S1: Image acquisition and preprocessing; Step S2: Semantic segmentation and feature adaptation includes: S201. The output panoramic image is input into the SegFormer-B2 semantic segmentation network based on the Transformer architecture. A cyclic padding mechanism is introduced in the network's convolutional layers and Transformer modules: that is, when processing the leftmost pixel of the image, the rightmost pixel of the image is used for padding convolution, and vice versa. The expression for filling the loop is as follows: ; : The filled feature tensor; W: Width of the panoramic image; Mod: Modulo operation, which enables logical connectivity between the left and right boundaries; S202. The model was pre-trained on the Cityscapes dataset and fine-tuned using a data augmentation algorithm for panoramic distortion; S203. Output: Output a semantic mask image with the same resolution as the original image. , where each pixel value Corresponding to a specific category ID; The semantic mask diagram is represented as follows: ； Network: refers to a deep neural network used to perform semantic segmentation tasks; ArgMax: Calculates the index of the maximum value; Step S3: Panoramic Monocular Depth Estimation and Scale Recovery: Step S3 involves inputting the panoramic image into the DINOV3 panoramic depth estimation network. This network employs a dual-branch structure, processing the ERP image and cubemap projection separately to generate a single-channel depth map. , where pixels The grayscale value directly corresponds to the straight-line distance of this point from the optical center of the camera. ; Geometric Scale Recovery: ; S: Absolute scaling factor; The mounting height of the camera's optical center relative to the ground; The pitch angle corresponding to the ground point; The relative depth value output by the deep network, i.e., the dimensionless value; Absolute depth map generation: ; : pixel The corresponding real-world straight-line distance; Step S4: Target detection and pixel extraction; Step S5: Solve spatial coordinates; Step S6: Based on time-series Kalman filter fusion, the final converged coordinates are output as the final geographical location of the feature.

2. The camera-based real-time semantic segmentation and geolocation method for street scene features according to claim 1, characterized in that, Step S1, image acquisition and preprocessing, includes: S101. Use an onboard multi-lens panoramic camera to collect video stream data around the vehicle, and at the same time obtain the vehicle's real-time latitude and longitude coordinates and three-axis attitude through a GNSS / IMU integrated navigation system; and use a hardware second pulse signal to trigger camera exposure to ensure that the exposure time of each frame of panoramic image is strictly aligned with the timestamp of the GNSS / IMU data. S102. The original images acquired from multiple lenses are stitched together and distortion corrected using a pre-calculated lookup table to generate a panoramic image with equidistant cylindrical projection. ； (u,v): pixel coordinates; : Output panoramic image pixel values; : The image of the k-th shot; , A pre-computed coordinate mapping lookup table maps panoramic pixels back to original lens pixels.

3. The camera-based real-time semantic segmentation and geolocation method for street scene features according to claim 2, characterized in that, Step S4, target detection and pixel extraction, consists of the following steps: S41. Perform connected component analysis on the mask graph output in step S2; S42. Extract all independent connected regions belonging to the target category; S43. Calculate the pixel coordinates of the geometric centroid or bottom center point of each connected region. : ; The set of pixels representing the k-th target in the panoramic image; The row coordinates of the bottom of the target; : Horizontal center column coordinates at the bottom of the target; S44. From the depth map Read the distance value corresponding to the coordinates: ; The straight-line distance from the root of the target to the optical center of the camera.

4. The camera-based real-time semantic segmentation and geolocation method for street scene features according to claim 3, characterized in that, Step S5 uses pixel coordinates, depth values, and camera pose to calculate the absolute coordinates of the target. The calculation process is as follows: Converting pixel coordinates to spherical coordinates: ； W, H: Width and height of the panoramic image; azimuth, range ; Pitch angle, range ; Spherical coordinates to camera coordinate system: ； : Three-dimensional coordinates in the camera system; Axial displacement: Camera system to vehicle engine system: Correcting the axial definition differences between the camera system and the vehicle system: ； : Three-dimensional coordinates in a machine system; definition: To the right of the camera's 3D coordinates For the front of the three-dimensional coordinates, The sky is in three-dimensional coordinates; Transformation logic: The "front" of the camera becomes the "front" of the car, and the "top" of the camera becomes the "top" of the car; Attitude compensation: Constructing a rotation matrix using IMU attitude angles : ； The target is offset eastward relative to the center of the vehicle. The target is offset northward relative to the center of the vehicle. The target is offset upwards relative to the center of the vehicle; The vehicle's current heading, pitch, and roll angles; Single-frame absolute coordinate calculation: Calculate the single-frame observation value of the target by combining the vehicle's GNSS coordinates: ； This refers to the Earth's average radius; : Target latitude and longitude measured in a single frame; : The vehicle's current latitude and longitude.

5. The camera-based real-time semantic segmentation and geolocation method for street scene features according to claim 4, characterized in that, Step S6, time-based Kalman filter fusion, consists of the following steps: S601. Using the shift algorithm, the object identified in the current frame is matched with the object in the historical frame tracking queue based on the proximity of positions; S602. Calculate the coordinates for each frame. As an input filter for the observed values; s603. Set the observation covariance matrix The closer the observation is to the camera, the higher its weight. S604. When the target moves out of the camera's field of view, the coordinates at which the output filter finally converges are taken as the final geographical location of the feature.

6. The camera-based real-time semantic segmentation and geolocation method for street scene features according to claim 5, characterized in that, State vector With observation vector : ; : Optimal position estimate after filtering; Virtual drift velocity, used to absorb systematic error trends; Range-adaptive observation noise : ； ； Standard deviation of reference measurement noise; Distance attenuation coefficient; Horizontal distance; Status Update: ; Kalman gain, with distance Increase Enlargement, leading to Decrease; Final result extraction: ; , : The final output coordinates.

7. A camera-based real-time semantic segmentation and geolocation system for street scene features, employing the camera-based real-time semantic segmentation and geolocation method for street scene features as described in any one of claims 1-6, characterized in that, It includes an image acquisition unit, a positioning and orientation unit, a calculation unit, and an output unit; Image acquisition unit: Uses a panoramic camera to acquire images; Positioning and attitude determination unit: A dual-antenna RTK-GNSS / IMU integrated navigation system is used for positioning and attitude determination measurements; Computational Unit: Utilizing panoramic depth estimation and spatial coordinate calculation algorithms, the three-dimensional spatial location of ground features is directly inferred from two-dimensional image pixels without prior map data, and the WGS84 absolute geographic coordinates of ground features are calculated in real time by combining GNSS / IMU data. By utilizing the Kalman filter algorithm and taking advantage of the multi-view observation brought about by vehicle motion, the noise of single-frame observation is automatically smoothed to output high-confidence geographic information data. Output unit: Visualizes the results.

8. The camera-based real-time semantic segmentation and geolocation system for street scenes according to claim 7, characterized in that, Image acquisition unit: Employs hard-triggered mode, the GNSS receiver sends q PPS signals per second to the camera trigger interface to force camera exposure, ensuring that the center exposure time of each panoramic image has an error of less than a set value compared to the timestamp of the GNSS / IMU data; Positioning and attitude determination unit: The IMU output frequency is set to 100Hz to provide the vehicle's real-time roll angle, pitch angle and yaw angle; Computing unit: Employs NVIDIA Jetson AGX Orin embedded computing module.

Images