A method and system for constructing a grid map based on binocular vision and semantic segmentation

Abstract

The application provides a kind of grid map construction method and system based on binocular vision and semantic segmentation, it is related to automatic driving technical field, the image data of target scene is collected by binocular camera in the application.The depth calculation and semantic segmentation are respectively executed to image data, and the three-dimensional point cloud data and semantic information corresponding to target scene are obtained.According to the three-dimensional point cloud data and semantic information corresponding to target scene, the three-dimensional space corresponding to target scene is divided into a plurality of grids, and the occupancy probability and semantic attribute of each grid are determined.According to the occupancy probability and semantic attribute of each grid, an initial grid map is generated.According to the dynamic target in the target scene, the occupancy probability and semantic attribute of each grid included in the initial grid map are updated, and a target grid map is obtained.The application can effectively improve the detection accuracy and applicability of the grid map while reducing the cost, and meet the use requirements in different use scenarios.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and system for constructing grid maps based on binocular vision and semantic segmentation. Background Technology

[0002] In intelligent mobility fields such as autonomous driving and robot navigation, environmental perception and map building are core technologies supporting the safe and efficient operation of equipment. Among them, grid maps, which can transform three-dimensional space into regular grid units and intuitively reflect the distribution of environmental obstacles by representing the occupancy probability of each grid, have become the mainstream environmental representation method. As the application scenarios of the technology expand from structured roads to complex urban roads, parks, and other multi-dynamic target scenarios, the industry is placing higher demands on the accuracy, cost control, and adaptability to dynamic scenarios of grid maps. It is necessary to accurately identify various targets such as road surfaces, vehicles, and pedestrians and locate their spatial positions, while controlling hardware costs to meet the needs of large-scale applications. Simultaneously, it is necessary to process environmental changes caused by the movement of dynamic targets in real time to avoid navigation decision errors due to map lag. This series of requirements presents grid map building technology with multi-dimensional performance balancing challenges.

[0003] To address these needs, three typical raster map construction schemes have emerged in existing technologies. Among them, the LiDAR-based method directly acquires 3D point cloud data by emitting laser beams and receiving reflected signals. The monocular vision-based method uses a low-cost single camera as its core, continuously acquiring environmental images to obtain rich texture information (such as object outlines and color details), achieving preliminary environmental perception based on image data. The semantic segmentation-based method uses a pre-trained deep learning model to classify image pixels, endowing the raster map with semantic attributes.

[0004] However, all three existing technical solutions have significant limitations, collectively leading to the most critical problem facing current grid map construction technology: the inability to simultaneously meet the comprehensive requirements of high precision, low cost, and accurate perception of small targets in dynamic scenes. Specifically, while LiDAR-based methods offer high depth accuracy, their hardware costs are high, far exceeding the cost tolerance of most scenarios. Furthermore, point cloud data exhibits sparsity, resulting in weak representation capabilities for small targets such as bicycles and traffic cones, and a delayed response to dynamic targets. Monocular vision-based methods, while low-cost and rich in texture, lack direct depth information, requiring indirect depth estimation through deep learning or geometric methods. In practical applications, this leads to a high depth error rate, making it difficult to accurately locate target spatial positions. Semantic segmentation-based methods, while possessing strong semantic recognition capabilities, only provide semantic information at the two-dimensional image level, failing to acquire the target's three-dimensional depth and accurately determine the target's actual distance and volume, resulting in errors in grid occupancy judgment. These inherent defects make existing technologies ill-suited for grid map construction in complex scenarios, easily impacting the navigation safety and practicality of smart devices due to inaccurate depth, misclassification, or excessive costs. Summary of the Invention

[0005] This invention provides a method and system for constructing grid maps based on binocular vision and semantic segmentation, which can effectively improve detection accuracy and applicability while reducing costs, and meet the needs of different application scenarios.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: Firstly, a method for constructing a raster map based on binocular vision and semantic segmentation is provided. This method includes: acquiring image data of a target scene using a binocular camera; performing depth calculation and semantic segmentation on the image data to obtain 3D point cloud data and semantic information corresponding to the target scene; the 3D point cloud data comprising a set of point clouds representing the 3D spatial locations of each object in the target scene, and the semantic information being the pixel category label corresponding to each pixel in the image data; dividing the 3D space of the target scene into multiple grids according to a preset grid size; fusing the 3D point cloud data and semantic information to obtain the pixel category label corresponding to each point cloud in the 3D point cloud data; determining the number of valid point clouds in each grid, where valid point clouds are those located within the left image range and whose depth values ​​are within the effective detection range; determining the occupancy probability and semantic attribute of each grid based on the number of valid point clouds and the pixel category label corresponding to each valid point cloud, where the semantic attribute of each grid is the pixel category label of the valid point cloud with the highest proportion in each grid; and generating an initial raster map based on the occupancy probability and semantic attribute of each grid. The target grid map is obtained by updating the occupancy probability and semantic attributes of each grid cell in the initial grid map based on the dynamic targets in the target scene.

[0007] The beneficial effects of this invention are as follows: The method provided by this invention acquires images through a binocular camera, then performs depth calculation and semantic segmentation, and subsequently generates a target grid map through grid division and probability attribute determination. Compared with related technologies, binocular vision combined with depth calculation can directly acquire 3D point cloud data, providing a reliable spatial location basis for grid occupancy probability calculation and ensuring basic accuracy. Simultaneously, the design of dynamically updating grid probabilities and attributes at a preset frequency breaks the limitation of existing static maps that cannot reflect environmental changes in real time, enabling the map to adapt to scenarios with dynamic target movement. In summary, the grid map construction core framework constructed by this invention, which balances low cost, basic environmental perception accuracy, and dynamic scene adaptability, can effectively improve detection accuracy and applicability while reducing costs, meeting the needs of different usage scenarios. Furthermore, the method provided by this invention divides the grid according to a preset size to adapt to the accuracy requirements of different scenarios. Secondly, data fusion binds semantic tags to each point cloud, avoiding the problem of point cloud and category classification. Thirdly, by filtering effective point clouds, the interference of invalid data on probability calculation can be reduced.

[0008] In one possible implementation of the first aspect, before performing depth calculation and semantic segmentation on the image data respectively, the method further includes: preprocessing the image data, the preprocessing including: denoising the image data of the target scene using a Gaussian filtering algorithm, correcting radial and tangential distortion of the denoised image data based on the calibration parameters of the binocular camera, and enhancing the contrast of the corrected image data using a histogram equalization algorithm.

[0009] This invention provides a method for image data preprocessing that proposes specific solutions for Gaussian filtering denoising, distortion correction, and histogram equalization, which can solve the problem of reduced accuracy in subsequent calculations due to incomplete or no preprocessing in related technologies. Noise in the original image can interfere with disparity map calculation, lens distortion can lead to spatial location mapping deviations, and low contrast can reduce the class discrimination of semantic segmentation. Therefore, the method provided by this invention can effectively remove image noise and reduce the interference of noise on depth calculation. Furthermore, radial and tangential distortion correction based on calibration parameters can effectively control errors and ensure accurate spatial correspondence of binocular images. Histogram equalization can improve image contrast, making it easier for semantic segmentation models to identify low-contrast targets such as road surfaces and traffic cones. Therefore, the method provided by this invention provides a high-quality data foundation for the accuracy of raster maps through preprocessing, avoiding the accumulation of map errors caused by defects in the original image.

[0010] In one possible implementation of the first aspect, the image data includes a left image and a right image of the target scene. Depth calculation and semantic segmentation are performed on the image data, including: determining a disparity map of the target scene based on the left and right images; converting the disparity map into 3D point cloud data based on basic information from the binocular cameras, including intrinsic and extrinsic parameters, where intrinsic parameters include focal length, principal point coordinates, and distortion coefficients, and extrinsic parameters include rotation matrix and translation vector; and classifying the pixels in the left image of the target scene using a pre-trained semantic segmentation model to obtain a pixel category label for each pixel.

[0011] The method provided by this invention determines the disparity map by comparing the left and right images and combines binocular intrinsic and extrinsic parameters to convert it into a 3D point cloud, which can effectively improve depth accuracy and ensure the accuracy of the target's spatial location. In terms of semantic segmentation, a pre-trained semantic segmentation model is used to classify the pixel categories of the left image and clearly define multiple different labels, which can improve the accuracy of semantic segmentation and meet the needs of different application scenarios.

[0012] In one possible implementation of the first aspect, the probability of occupation The formula for determining it is: .

[0013] Where n is the number of valid point clouds, W i Let be the semantic weight coefficient corresponding to the i-th valid point cloud, and each pixel category label corresponds to a different semantic weight coefficient.

[0014] The method provided by this invention calculates the occupancy probability by semantic weight weighted average and assigns attributes according to the semantic label with the highest proportion in the grid. This occupancy probability can more accurately reflect the degree of influence of different types of targets on grid occupancy, and the semantic attributes can accurately correspond to the core targets of the grid.

[0015] In one possible implementation of the first aspect, the 3D point cloud data and semantic information are fused to obtain a pixel category label corresponding to each point cloud in the 3D point cloud data. This includes: establishing a projection matrix based on the principle of perspective projection and the intrinsic parameters of a binocular camera; projecting each point cloud in the 3D point cloud data onto the left image plane and matching the pixel position of each point cloud in the left image; and determining the pixel category label of each point cloud based on the semantic information to obtain the pixel category label corresponding to each point cloud in the 3D point cloud data.

[0016] The method provided by this invention refines the specific process of fusing 3D point clouds with semantic information. By accurately projecting each point cloud onto the left image plane and matching pixel positions, it ensures that each point cloud corresponds to a unique pixel position. Then, based on this pixel position, a label is obtained from the semantic information, achieving precise binding between point clouds and category labels. This ensures that the semantic labels of the point clouds within each grid are accurate and reliable, further improving the accuracy of the grid map.

[0017] In one possible implementation of the first aspect, after determining the occupancy probability and semantic attributes of each grid cell based on the number of valid point clouds and the pixel category label corresponding to each valid point cloud, the method further includes: updating the occupancy probability of each grid cell using a Bayesian update rule.

[0018] The Bayesian update rule is: .

[0019] in, The updated occupancy probability. To occupy a probability in history, This represents the current occupancy probability.

[0020] The method provided by this invention combines historical occupancy probabilities with current observed occupancy probabilities through Bayesian rules for iterative updates. It can use historical data to correct single-frame anomalies, reduce single-frame noise interference, and make the grid occupancy probability more consistent with the long-term stable state of the environment. This avoids map fluctuations caused by single-frame data anomalies, improves the stability and reliability of grid maps, and provides more continuous environmental data for navigation decisions.

[0021] In one possible implementation of the first aspect, the method further includes: increasing the semantic weight coefficients corresponding to the pixel category labels for pedestrians, bicycles, and traffic cones when the vehicle density or pedestrian density of the target scene is greater than or equal to a preset vehicle density or pedestrian density. Decreasing the semantic weight coefficients corresponding to each pixel category label when the standard deviation of all point clouds in the target scene is greater than a preset standard deviation threshold. Adjusting the semantic weight coefficients corresponding to the pixel category labels of any target when the localization deviation rate of any target in the target scene is greater than or equal to a preset deviation rate threshold, wherein the localization deviation rate is determined based on historical continuous multi-frame image data of the target scene.

[0022] The method provided by this invention increases the weights of pedestrians, bicycles, and traffic cones in complex scenarios where vehicle density is greater than or equal to a preset vehicle density or pedestrian density is greater than or equal to a preset pedestrian density. This avoids underweighting of small targets due to occlusion by large targets, leading to misjudgments of occupancy probability. Secondly, it reduces the weights of corresponding categories when the standard deviation of point cloud depth is large, minimizing interference from low-precision data in probability calculations. Finally, based on feedback from historical continuous multi-frame data, the weights are adjusted through Bayesian optimization, effectively reducing target localization errors. This dynamic weighting mechanism allows semantic weights to adapt in real-time to scene, data quality, and historical biases, avoiding the limitations of fixed weights and further improving the accuracy of grid occupancy probability calculations, thus adapting to diverse and complex scenarios.

[0023] In one possible implementation of the first aspect, the occupancy probability and semantic attributes of each grid cell in the initial grid map are updated based on dynamic targets in the target scene to obtain a target grid map. This includes: for each grid cell, determining the difference in occupancy probability between adjacent time points based on historical consecutive frames of the target scene. Grid cells with occupancy probability difference values ​​greater than or equal to a preset threshold are identified as potential dynamic grid cells. Cluster analysis is performed on the potential dynamic grid cells to obtain a candidate set of dynamic targets. The candidate set includes at least one grid cell cluster, each grid cell cluster includes one or more potential dynamic grid cells, and each grid cell cluster corresponds to a dynamic target. Based on the Kalman filter algorithm, prediction is performed based on the six-dimensional state vector of each dynamic target corresponding to historical consecutive frames to obtain the predicted position corresponding to each dynamic target. The predicted position corresponding to each dynamic target is mapped to each grid cell in the initial grid map to determine the updated occupancy probability and semantic attributes of each grid cell.

[0024] Among them, the updated occupancy probability The formula for determining it is: .

[0025] The occupancy probability corresponding to the initial grid map. The occupancy probability is determined based on the predicted location corresponding to each dynamic target. This is the weighting coefficient. Furthermore, if the updated occupancy probability is greater than 1, the updated occupancy probability is set to 1; if the updated occupancy probability is less than 0, the updated occupancy probability is set to 0.

[0026] The initial raster map is updated based on the updated occupancy probability and semantic attributes of each raster to obtain the target raster map.

[0027] Before mapping the predicted location of each dynamic target to each grid cell in the initial grid map and determining the updated occupancy probability and semantic attributes of each grid cell, the method further includes determining a weighting coefficient based on the motion characteristics of each dynamic target. Specifically, when the speed of the dynamic target is greater than a first speed threshold, the weighting coefficient is a first value; when the speed of the dynamic target is less than a second speed threshold, the weighting coefficient is a second value, and the first value is less than the second value. Furthermore, when the speed standard deviation of the dynamic target across multiple consecutive frames is greater than a speed standard deviation threshold, the weighting coefficient is determined based on the speed standard deviation and is negatively correlated with it.

[0028] The method provided by this invention refines the dynamic target processing and raster update process. By comparing raster probability differences across multiple consecutive historical images, it identifies potential dynamic rasters and forms a dynamic target candidate set, enabling rapid capture of dynamic targets. Secondly, based on Kalman filtering to predict target positions, it achieves accurate prediction of dynamic targets. By weighted probability updates and dynamically adjusting weight coefficients according to target speed, it avoids the problems of lag in fast targets and instability in slow targets caused by fixed weight coefficients. In summary, the method provided by this invention achieves rapid detection, accurate prediction, and adaptive updating of dynamic targets, enabling maps to reflect dynamic target movement in real time and reducing navigation safety risks caused by dynamic target lag.

[0029] In one possible implementation of the first aspect, after updating the occupancy probability and semantic attributes of each grid cell in the initial grid map based on dynamic targets in the target scene to obtain the target grid map, the method further includes: smoothing the target grid map using a Gaussian filtering algorithm, wherein the kernel size of the Gaussian filtering algorithm is determined according to the preset grid size of the target grid map; and acquiring real-time image data of the target scene at a preset frequency using a binocular camera, and updating the target grid map based on the real-time image data.

[0030] The method provided by this invention, through Gaussian filtering smoothing and high-frequency real-time updates, can reduce probability abrupt changes caused by point cloud noise and dynamic target prediction errors, making the probability transition between grids more natural, improving map smoothness, and avoiding interference with subsequent navigation decisions due to probability abrupt changes. Secondly, by updating the map in real time at a preset frequency, it ensures that the map can reflect environmental changes in real time.

[0031] Secondly, this invention provides a grid map construction system based on binocular vision and semantic segmentation, applicable to any possible implementation of the grid map construction method based on binocular vision and semantic segmentation in the first aspect described above. The system includes: an image acquisition module for acquiring image data of a target scene using a binocular camera; an image processing module for performing depth calculation and semantic segmentation on the image data to obtain 3D point cloud data and semantic information corresponding to the target scene, wherein the 3D point cloud data includes a set of point clouds representing the 3D spatial positions of each object in the target scene, and the semantic information is the pixel category label corresponding to each pixel in the image data; a grid processing module for dividing the 3D space of the target scene into multiple grids according to a preset grid size; a fusion processing module for fusing the 3D point cloud data and semantic information to obtain the pixel category label corresponding to each point cloud in the 3D point cloud data; and a determination of the number of valid point clouds in each of the multiple grids, wherein valid point clouds are point clouds located within the left image range and whose depth values ​​are within the effective detection range. It also determines the occupancy probability and semantic attributes of each raster based on the number of valid point clouds and the pixel category label corresponding to each valid point cloud. The semantic attribute of each raster is the pixel category label of the valid point cloud with the highest proportion in each raster. The map generation module generates an initial raster map based on the occupancy probability and semantic attributes of each raster. The map update module updates the occupancy probability and semantic attributes of each raster included in the initial raster map based on dynamic targets in the target scene, obtaining the target raster map.

[0032] Understandably, the beneficial effects that the system of the second aspect described above can achieve can be referenced from the beneficial effects of the first aspect and any of its possible design methods, which will not be repeated here. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a raster map construction method based on binocular vision and semantic segmentation provided in an embodiment of the present invention; Figure 3 A flowchart illustrating another method for constructing a raster map based on binocular vision and semantic segmentation, provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a grid map construction system provided in an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. In the description of the present invention, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. The "or" in the present invention is merely a description of the relationship between the related objects, indicating that three relationships can exist. For example, A or B can represent: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Furthermore, in the description of the present invention, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.

[0035] Furthermore, to facilitate a clear description of the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0036] In this embodiment of the invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this embodiment of the invention should not be construed as superior or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0037] In intelligent mobility fields such as autonomous driving and robot navigation, environmental perception and map building are core technologies supporting the safe and efficient operation of equipment. Among them, grid maps, which can transform three-dimensional space into regular grid units and intuitively reflect the distribution of environmental obstacles by representing the occupancy probability of each grid, have become the mainstream environmental representation method. As the application scenarios of the technology expand from structured roads to complex urban roads, parks, and other multi-dynamic target scenarios, the industry is placing higher demands on the accuracy, cost control, and adaptability to dynamic scenarios of grid maps. It is necessary to accurately identify various targets such as road surfaces, vehicles, and pedestrians and locate their spatial positions, while controlling hardware costs to meet the needs of large-scale applications. Simultaneously, it is necessary to process environmental changes caused by the movement of dynamic targets in real time to avoid navigation decision errors due to map lag. This series of requirements presents grid map building technology with multi-dimensional performance balancing challenges.

[0038] To address these needs, three typical raster map construction schemes have emerged in existing technologies. Among them, the LiDAR-based method directly acquires 3D point cloud data by emitting laser beams and receiving reflected signals. The monocular vision-based method uses a low-cost single camera as its core, continuously acquiring environmental images to obtain rich texture information (such as object outlines and color details), achieving preliminary environmental perception based on image data. The semantic segmentation-based method uses a pre-trained deep learning model to classify image pixels, endowing the raster map with semantic attributes.

[0039] However, all three existing technical solutions have significant limitations, collectively leading to the most critical problem facing current grid map construction technology: the inability to simultaneously meet the comprehensive requirements of high precision, low cost, and accurate perception of small targets in dynamic scenes. Specifically, while LiDAR-based methods offer high depth accuracy, their hardware costs are high, far exceeding the cost tolerance of most scenarios. Furthermore, point cloud data exhibits sparsity, resulting in weak representation capabilities for small targets such as bicycles and traffic cones, and a delayed response to dynamic targets. Monocular vision-based methods, while low-cost and rich in texture, lack direct depth information, requiring indirect depth estimation through deep learning or geometric methods. In practical applications, this leads to a high depth error rate, making it difficult to accurately locate target spatial positions. Semantic segmentation-based methods, while possessing strong semantic recognition capabilities, only provide semantic information at the two-dimensional image level, failing to acquire the target's three-dimensional depth and accurately determine the target's actual distance and volume, resulting in errors in grid occupancy judgment. These inherent defects make existing technologies ill-suited for grid map construction in complex scenarios, easily impacting the navigation safety and practicality of smart devices due to inaccurate depth, misclassification, or excessive costs.

[0040] In view of this, embodiments of the present invention provide a method and system for constructing a grid map based on binocular vision and semantic segmentation. The method includes: acquiring image data of a target scene using a binocular camera; performing depth calculation and semantic segmentation on the image data to obtain 3D point cloud data and semantic information corresponding to the target scene; the 3D point cloud data includes a set of point clouds corresponding to the 3D spatial positions of each object in the target scene, and the semantic information is the pixel category label corresponding to each pixel in the image data; dividing the 3D space of the target scene into multiple grids according to a preset grid size; fusing the 3D point cloud data and semantic information to obtain the pixel category label corresponding to each point cloud in the 3D point cloud data; determining the number of effective point clouds in each grid, wherein effective point clouds are those located within the left image range and whose depth values ​​are within the effective detection range; determining the occupancy probability and semantic attribute of each grid based on the number of effective point clouds and the pixel category label corresponding to each effective point cloud, wherein the semantic attribute of each grid is the pixel category label of the most prevalent effective point cloud in each grid; and generating an initial grid map based on the occupancy probability and semantic attribute of each grid. The target grid map is obtained by updating the occupancy probability and semantic attributes of each grid cell in the initial grid map based on the dynamic targets in the target scene.

[0041] The method provided by this invention acquires images using a binocular camera, then performs depth calculation and semantic segmentation. Following this, it generates a target grid map through grid division and probability attribute determination. Compared to related technologies, binocular vision combined with depth calculation can directly acquire 3D point cloud data, providing a reliable spatial basis for grid occupancy probability calculation and ensuring basic accuracy. Simultaneously, the design of dynamically updating grid probabilities and attributes at a preset frequency overcomes the limitation of existing static maps that cannot reflect environmental changes in real time, enabling the map to adapt to scenarios with dynamic target movement. In summary, the grid map construction core framework constructed by this invention, which balances low cost, basic environmental perception accuracy, and dynamic scene adaptability, can effectively improve detection accuracy and applicability while reducing costs, meeting the needs of different usage scenarios. Furthermore, the method provided by this invention divides the grid according to a preset size to adapt to the accuracy requirements of different scenarios. Secondly, data fusion binds semantic tags to each point cloud, avoiding the problem of point cloud and category classification. Thirdly, by filtering effective point clouds, the interference of invalid data on probability calculation can be reduced.

[0042] In some embodiments, the grid map construction method based on binocular vision and semantic segmentation provided in this invention can be executed by a grid map construction system 100 based on binocular vision and semantic segmentation (hereinafter referred to as grid map construction system 100).

[0043] As an example, the raster map building system 100 can be any electronic device 200 with data processing capabilities, such as a general-purpose computer, personal computer, laptop computer, switch, or tablet computer. The specific implementation of the raster map building system 100 is not limited here.

[0044] Figure 1 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention is shown. The electronic device 200 includes a processor 210, a memory 220, and a communication interface 230.

[0045] Processor 210 may include one or more processing cores. Processor 210 connects to various parts within electronic device 200 using various interfaces and lines, and performs various functions and processes data of electronic device 200 by running or executing instructions, programs, code sets, or instruction sets stored in memory 220, and by calling data stored in memory 220. Optionally, processor 210 may be implemented using at least one of the following hardware forms: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA).

[0046] The memory 220 may include random access memory (RAI) or read-only memory (ROI). Optionally, the memory 220 may include non-transitory computer-readable storage ledger. The memory 220 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 220 may include a stored program area. The stored program area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the various method embodiments described above, etc.

[0047] Communication interface 230 is used to communicate with other devices, equipment or communication networks, such as data storage devices, image processing devices or Ethernet, wireless access network (RAN), wireless local area network (WLAN), etc.

[0048] In terms of physical implementation, the aforementioned devices (such as processor 210, memory 220, and communication interface 230) can each be devices within the same device (such as a laptop computer). Alternatively, at least two of these devices can be located within the same device, i.e., as different devices within the same device, similar to the deployment of devices or components in a distributed system.

[0049] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments of the present invention, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0050] The following description, in conjunction with the accompanying drawings, illustrates a raster map construction method based on binocular vision and semantic segmentation provided by an embodiment of the present invention.

[0051] Figure 2 This is a flowchart illustrating a raster map construction method based on binocular vision and semantic segmentation, provided as an embodiment of the present invention. Optionally, this method can be... Figure 1 The illustrated electronic device 200 performs this operation. The method may include the following steps: S1. Acquire image data of the target scene using a binocular camera.

[0052] For example, the grid map construction system 100 acquires scene image data (left and right images) in real time using a binocular camera, with the acquisition frame rate set to 25-30 frames per second. The method provided by this invention acquires environmental visual information through a binocular camera, providing raw data for subsequent processing. Compared to LiDAR, it effectively reduces costs while ensuring the continuity of data acquisition.

[0053] S2. Perform depth calculation and semantic segmentation on the image data respectively to obtain the 3D point cloud data and semantic information corresponding to the target scene.

[0054] Specifically, the 3D point cloud data includes a set of point clouds representing the 3D spatial location of each object in the target scene, and the semantic information is the pixel category label corresponding to each pixel in the image data.

[0055] In one possible implementation, the image data includes a left image and a right image of the target scene. The aforementioned S2 includes: The disparity map of the target scene is determined based on the left and right images. Using basic information from the binocular cameras, the disparity map is converted into 3D point cloud data. This basic information includes intrinsic and extrinsic parameters. Intrinsic parameters include focal length, principal point coordinates, and distortion coefficients; extrinsic parameters include rotation matrix and translation vector. A pre-trained semantic segmentation model is used to classify the pixels in the left image of the target scene, obtaining the pixel category label for each pixel.

[0056] In one possible implementation, pixel category labels include road surface, lane lines, vehicles, pedestrians, bicycles, traffic cones, and background. It should be understood that the above pixel category labels are merely illustrative, and the embodiments of the present invention do not impose any particular limitation on the specific implementation of the pixel category labels.

[0057] Specifically, the raster map construction system 100 uses semi-global matching or deep learning methods to calculate disparity maps on preprocessed image data (left and right images), effectively reducing disparity calculation errors. Then, based on the intrinsic parameters (including focal length, principal point coordinates, and distortion coefficients, obtained through Zhang's calibration method) and extrinsic parameters (including rotation matrix and translation vector, obtained through stereo calibration) of the binocular camera, the disparity map is converted into 3D point cloud data, effectively reducing the depth calculation error rate and achieving the conversion from 2D images to 3D spatial information. Compared to depth estimation using monocular vision, this significantly improves accuracy.

[0058] Furthermore, the grid map construction system 100 utilizes pre-trained semantic segmentation models (such as DeepLabv3+, SegFormer, etc.) to perform semantic segmentation on the left image, classifying each pixel in the image into different semantic categories (semantic categories include road surface, lane lines, vehicles, pedestrians, bicycles, traffic cones, background, etc.), which can effectively improve the accuracy of semantic segmentation and thus obtain the semantic information of the image. Existing technologies mostly perform single-target segmentation; this step, through multi-category fine segmentation, lays the foundation for subsequent semantic fusion and improves the ability to recognize small targets.

[0059] As described above, the method provided in this embodiment of the invention, by determining the disparity map using the left and right images and combining binocular intrinsic and extrinsic parameters to convert it into a 3D point cloud, can effectively improve depth accuracy and ensure the accuracy of the target's spatial location. In terms of semantic segmentation, by using a pre-trained semantic segmentation model to classify the pixel categories of the left image and clearly identifying multiple different labels, the accuracy of semantic segmentation can be improved, meeting the needs of different application scenarios.

[0060] In some embodiments, prior to S2 above, the method provided by the present invention further includes: The image data is preprocessed, including: using a Gaussian filtering algorithm to denoise the image data of the target scene, correcting radial and tangential distortion of the denoised image data based on the calibration parameters of the binocular camera, and using a histogram equalization algorithm to enhance the contrast of the corrected image data.

[0061] In one example, the raster map construction system 100 performs denoising (using a Gaussian filter algorithm with a 3×3 kernel size), correction (correcting radial and tangential distortion based on camera calibration parameters), and enhancement (using a histogram equalization algorithm to improve image contrast) on the acquired image data. This preprocessing effectively removes image noise, reduces distortion correction errors, and improves image contrast, providing a higher-quality data foundation for depth calculations and semantic segmentation.

[0062] This invention provides a method for image data preprocessing that proposes specific solutions for Gaussian filtering denoising, distortion correction, and histogram equalization, which can solve the problem of reduced accuracy in subsequent calculations due to incomplete or no preprocessing in related technologies. Noise in the original image can interfere with disparity map calculation, lens distortion can lead to spatial location mapping deviations, and low contrast can reduce the class discrimination of semantic segmentation. Therefore, the method provided by this invention can effectively remove image noise and reduce the interference of noise on depth calculation. Furthermore, radial and tangential distortion correction based on calibration parameters can effectively control errors and ensure accurate spatial correspondence of binocular images. Histogram equalization can improve image contrast, making it easier for semantic segmentation models to identify low-contrast targets such as road surfaces and traffic cones. Therefore, the method provided by this invention provides a high-quality data foundation for the accuracy of raster maps through preprocessing, avoiding the accumulation of map errors caused by defects in the original image.

[0063] S3. Divide the three-dimensional space of the target scene into multiple grids according to the preset grid size; perform fusion processing on the three-dimensional point cloud data and semantic information to obtain the pixel category label corresponding to each point cloud included in the three-dimensional point cloud data; determine the number of effective point clouds included in each grid in the multiple grids, and determine the occupancy probability and semantic attributes of each grid according to the number of effective point clouds and the pixel category label corresponding to each effective point cloud.

[0064] Among them, the effective point cloud is the point cloud located within the left image range and whose depth value is within the effective detection range, and the semantic attribute of each grid is the pixel category label of the effective point cloud with the highest proportion in each grid.

[0065] In one example, the grid map building system 100 divides the three-dimensional space of the target scene into regular three-dimensional grids (or selects two-dimensional grids according to application requirements) according to a set grid size (the grid size can be adjusted according to application needs, such as 0.1m×0.1m×0.1m for autonomous driving scenarios and 0.05m×0.05m×0.05m for robot navigation scenarios), thus building the basic framework of the map.

[0066] Furthermore, the 3D point cloud data and semantic information are fused to obtain the pixel category label corresponding to each point in the 3D point cloud data, including: A projection matrix is ​​established based on the principle of perspective projection and the intrinsic parameters of a binocular camera. Each point cloud in the 3D point cloud data is projected onto the left image plane, and the corresponding pixel position of each point cloud in the left image is matched. The pixel category label of each point cloud is determined based on semantic information, thus obtaining the pixel category label corresponding to each point cloud included in the 3D point cloud data.

[0067] Specifically, the raster map construction system 100 projects 3D point cloud data onto the left image plane (using the principle of perspective projection, establishing a projection matrix based on camera intrinsic parameters) to find the corresponding pixel position of each point cloud point in the image. Based on the pixel position, the corresponding semantic labels (person, car, truck, tricycle, traffic cone, etc.) are obtained from the semantic segmentation results and assigned to the corresponding point cloud point, realizing the fusion of depth information and semantic information, and assigning semantic attributes to each point cloud point. Compared with the existing multi-sensor fusion which "ignores the precise matching of semantics and depth", this step can effectively improve the fusion accuracy through pixel-level label assignment. Finally, invalid point filtering is performed to remove invalid point cloud points that are outside the image range or have abnormal depth values ​​(such as depth values ​​less than 0 or greater than the effective detection distance of the camera, which is set to 0.5-50 meters), which can improve the invalid point filtering rate and ensure the validity and accuracy of the fused data.

[0068] In one possible implementation, the probability of occupancy... The formula for determining it is: .

[0069] Where n is the number of valid point clouds, W i Let be the semantic weight coefficient corresponding to the i-th valid point cloud, and each pixel category label corresponds to a different semantic weight coefficient.

[0070] The method provided by this invention calculates the occupancy probability by semantic weight weighted average and assigns attributes according to the semantic label with the highest proportion in the grid. This occupancy probability can more accurately reflect the degree of influence of different types of targets on grid occupancy, and the semantic attributes can accurately correspond to the core targets of the grid.

[0071] In one example, the number of point clouds and semantic categories in each grid are counted, different weight coefficients are set based on the semantic categories, and the initial occupancy probability of the grid is calculated according to the distribution of the number of point clouds and semantic categories.

[0072] The initial occupancy probability is calculated using the following formula: ,in The initial occupancy probability of the grid. is the semantic weight coefficient corresponding to the i-th point cloud point, and n is the number of point clouds in the raster.

[0073] Furthermore, the raster map construction system 100 assigns corresponding semantic attributes to each raster based on the semantic labels of the point clouds within the raster (the semantic label with the highest proportion within the raster is used as the semantic attribute of the raster), generating a raster map with semantic information, which can effectively improve the accuracy of semantic attribute assignment.

[0074] Optionally, the method provided in this embodiment of the invention further includes: If the vehicle density or pedestrian density in the target scene is greater than or equal to a preset vehicle density or pedestrian density, increase the semantic weight coefficients corresponding to the pixel category labels for pedestrians, bicycles, and traffic cones. If the standard deviation of all point clouds in the target scene is greater than a preset standard deviation threshold, decrease the semantic weight coefficients corresponding to each pixel category label. If the localization error rate of any target in the target scene is greater than or equal to a preset error rate threshold, adjust the semantic weight coefficients corresponding to the pixel category labels of any target based on the localization error rate, where the localization error rate is determined based on historical continuous multi-frame image data of the target scene.

[0075] In one example, the preset vehicle density is 5 vehicles / 100m. 2 The preset pedestrian density is 10 people / 100m. 2 .

[0076] In one example, the weighting coefficients are set using a dynamic adjustment mechanism, taking into account both the inherent attributes of semantic categories and real-time environmental conditions: initial values ​​are set based on the inherent influence of semantic categories on the occupied space. For example: rigid dynamic targets such as vehicles and pedestrians: basic weighting coefficients are set to 0.8-0.9. Small static targets such as bicycles and traffic cones: basic weighting coefficients are set to 0.6-0.7. Passable areas such as road surfaces and lane lines: basic weighting coefficients are set to 0.1-0.2. Background areas: basic weighting coefficients are set to 0.05-0.1.

[0077] Based on environmental complexity: When dense traffic flow is detected (vehicle density ≥ 5 vehicles / 100m) 2 Pedestrian gathering (pedestrian density ≥ 10 people / 100m) 2In complex scenarios such as bicycles, the weight of small targets (such as bicycles) is automatically increased to 0.7-0.8 to avoid misjudgment due to occlusion by large targets, thus effectively improving the recognition accuracy of small targets.

[0078] Based on sensor confidence: Through error analysis of binocular vision depth calculation (calculating the standard deviation of depth values; if the standard deviation is ≥0.5 meters, it is judged as low depth accuracy), for areas with low depth accuracy (such as distant targets, distance ≥30 meters), the weight of their corresponding semantic category is reduced (e.g., the weight of vehicles is reduced to 0.6-0.7) to reduce the interference of low-precision data on the occupancy probability calculation, thereby reducing the occupancy probability error rate.

[0079] Based on historical data feedback: If a certain semantic target (such as a traffic cone) frequently shows positioning deviation in past scenarios (statistics of the past 100 frames of data, if the proportion of frames with positioning deviation ≥50cm is ≥30%, it is judged as frequent deviation), its weight coefficient is dynamically adjusted through Bayesian optimization algorithm (for every 10% increase in deviation rate, the weight increases by 5%-8%), so as to reduce the target positioning error.

[0080] In this way, the method provided by this invention increases the weights of pedestrians, bicycles, and traffic cones in complex scenarios where vehicle density is greater than or equal to a preset vehicle density or pedestrian density is greater than or equal to a preset pedestrian density, avoiding the underestimation of the weights and misjudgment of occupancy probability caused by small targets being occluded by large targets. Secondly, it reduces the weights of corresponding categories when the standard deviation of point cloud depth is large, reducing the interference of low-precision data on probability calculation. Finally, based on feedback from historical continuous multi-frame data, the weights are adjusted through Bayesian optimization, effectively reducing the target localization error. This dynamic weighting mechanism allows semantic weights to adapt in real time to scene, data quality, and historical deviations, avoiding the limitations of fixed weights, further improving the accuracy of grid occupancy probability calculation, and adapting to diverse and complex scenarios.

[0081] In one possible implementation, after S3 above, the method further includes: The occupancy probability of each grid cell is updated using a Bayesian update rule.

[0082] The Bayesian update rule is: .

[0083] in, The updated occupancy probability. To occupy a probability in history, This represents the current occupancy probability.

[0084] As can be seen from the above, the method provided by the embodiments of the present invention combines the historical occupancy probability with the current observed occupancy probability through Bayesian rules for iterative updates. It can use historical data to correct single-frame anomalies, reduce single-frame noise interference, make the grid occupancy probability more consistent with the long-term stable state of the environment, avoid map fluctuations caused by single-frame data anomalies, improve the stability and reliability of the grid map, and provide more continuous environmental data for navigation decisions.

[0085] S4. Generate an initial raster map based on the occupancy probability and semantic attributes of each raster.

[0086] S5. Update the occupancy probability and semantic attributes of each grid cell in the initial grid map based on the dynamic targets in the target scene to obtain the target grid map.

[0087] In some embodiments, see Figure 3 The above S5 includes: S51. For each grid cell, determine the occupancy probability difference value of each grid cell at adjacent time points based on the historical consecutive frames of the target scene.

[0088] Specifically, by comparing multiple consecutive frames (usually 3-5 frames) of raster maps, the difference in the occupancy probability of each raster at adjacent times is calculated.

[0089] The formula for determining the probability difference value is: .

[0090] in, This represents the difference in occupancy probability of grid g. Let g be the occupancy probability of grid g at time t. $ represents the occupancy probability of grid g at time t-1.

[0091] Set difference threshold =0.3), when > When the grid is identified as a potential dynamic grid, it is marked as such. The DBSCAN density clustering algorithm (with a cluster radius of 0.3-0.5 meters and a minimum number of cluster points of 3-5) is used to perform cluster analysis on the potential dynamic grids. Based on the spatial proximity and state change trends of the grids, grids that meet the conditions are aggregated into clusters. Each cluster corresponds to a potential dynamic target, forming a dynamic target candidate set, which improves the accuracy of dynamic target detection and reduces detection latency.

[0092] S52. Grids with an occupancy probability difference value greater than or equal to a preset occupancy probability difference value threshold are identified as potential dynamic grids.

[0093] Based on the above examples, set a difference threshold. =0.3, when > When this happens, the raster is marked as a potential dynamic raster.

[0094] S53. Perform cluster analysis on the potential dynamic grid to obtain a set of dynamic target candidates.

[0095] The dynamic target candidate set includes at least one grid cluster, each grid cluster includes one or more potential dynamic grids, and each grid cluster corresponds to a dynamic target.

[0096] S54. Based on the Kalman filter algorithm, the predicted position of each dynamic target is obtained by predicting the six-dimensional state vector of each dynamic target corresponding to multiple consecutive historical frames.

[0097] Specifically, a six-dimensional state vector is defined for each dynamic objective. , where x, y, and z are the x-axis, y-axis, and z-axis position coordinates of the target in three-dimensional space (unit: meters). These represent the target's velocities (in meters per second) along the x, y, and z axes, respectively, indicating the target's three-dimensional position and velocity.

[0098] The state of a dynamic target is estimated by using a Kalman filter (Kalman filtering algorithm) through two stages: prediction and update.

[0099] Prediction phase: Based on the state of the previous moment and the state transition matrix F (the state transition matrix F is a 6×6 matrix, constructed based on the uniform motion model, in the form of...) .

[0100] in, (The time interval between adjacent moments, i.e., the frame interval, in seconds), is used to calculate the prior state estimate for the current moment. (formula: ), and update the covariance matrix. (formula: ),in Let be the prior covariance matrix at time t. Let be the posterior covariance matrix at time t-1. Let F be the transpose of the state transition matrix, and Q be the process noise covariance matrix, which is set based on actual motion noise statistics and is usually a diagonal matrix with diagonal elements ranging from 1 to 2. ).

[0101] Update phase: Acquiring new observation data (The observed data is the target state calculated from the current frame image, including position and velocity.) Then, the Kalman gain is calculated using the observation matrix H (the observation matrix H is a 6×6 identity matrix, since the dimensions of the observed data and the state vector are the same). (formula: ),in Let be the Kalman gain at time t. Let H be the transpose of the observation matrix, and R be the observation noise covariance matrix, which is defined based on observation error statistics and is usually a diagonal matrix with diagonal elements ranging from 1 to 2. ), thereby updating the predicted state (formula: and covariance matrix (formula: (where I is a 6×6 identity matrix), which enables the prediction of the future position of dynamic targets, effectively reducing position prediction error and velocity estimation error.

[0102] S55. Map the predicted location of each dynamic target to each grid cell in the initial grid map, and determine the updated occupancy probability and semantic attributes of each grid cell.

[0103] Among them, the updated occupancy probability The formula for determining it is: .

[0104] This represents the occupancy probability (initial occupancy probability) corresponding to the initial grid map. The occupancy probability (predicted occupancy probability) is determined based on the predicted location corresponding to each dynamic target. This is the weighting coefficient. Furthermore, if the updated occupancy probability is greater than 1, the updated occupancy probability is set to 1; if the updated occupancy probability is less than 0, the updated occupancy probability is set to 0.

[0105] The value range of is [0,1]. The value range is also [0,1].

[0106] Specifically, the Kalman filter outputs the future position and velocity information of the dynamic target, which is used to determine the grid range that the target may occupy in the next 1-3 frames (the movement distance is calculated based on the velocity and frame interval to determine the grid coverage). By transforming the predicted 3D position coordinates into the coordinate system of the grid map (calculating the corresponding grid index based on the grid size), the corresponding grid set G is accurately located, providing a basis for updating the occupancy probability of these grids. Compared with existing technologies that "update only based on the current frame", predicting the position of dynamic targets in advance can effectively improve the map's adaptability to dynamic scenes.

[0107] S56. Update the initial grid map based on the updated occupancy probability and semantic attributes of each grid cell to obtain the target grid map.

[0108] Optionally, the method provided in this embodiment of the invention further includes: The weighting coefficients are determined based on the motion characteristics of each dynamic target. Specifically, when the speed of the dynamic target is greater than a first speed threshold, the weighting coefficient is the first value; when the speed of the dynamic target is less than a second speed threshold, the weighting coefficient is the second value, and the first value is less than the second value. When the speed standard deviation of the dynamic target in multiple consecutive frames is greater than a speed standard deviation threshold, the weighting coefficient is determined based on the speed standard deviation and is negatively correlated with the speed standard deviation.

[0109] The following example illustrates how to adjust the weighting coefficients.

[0110] Weighting coefficients are dynamically adjusted as follows: The specific value of α is determined based on the motion characteristics of the dynamic target: When the target moves at a high speed (e.g., speed greater than 5 m / s, calculated from the velocity components in the six-dimensional state vector), its position changes rapidly, and the occupancy probability based on the predicted position has higher reference value. In this case, the value of α is smaller, for example, 0.3-0.5, to refer more to the predicted data and reduce the lag problem of occupancy probability caused by the rapid movement of the target. When the target moves at a slow speed (e.g., speed less than 1 m / s), the static occupancy probability better reflects the actual situation, and the value of α is larger, for example, 0.6-0.8, prioritizing the reference to the static data of the current frame to ensure the stability of the occupancy probability. For targets with unstable motion states (e.g., speed fluctuation standard deviation greater than 2 m / s, determined by calculating the standard deviation of speed data for 5-10 consecutive frames), an adaptive adjustment strategy is adopted: if the speed fluctuation shows an increasing trend, the value of α decreases linearly with the increase of the fluctuation amplitude (α decreases by 0.05-0.1 for every 0.5 m / s increase in the fluctuation standard deviation). If the fluctuation shows a decreasing trend, the α value gradually rises back to around 0.5. The α value is then finely adjusted in real time based on the target's recent movement trend to ensure the accuracy of the occupancy probability update.

[0111] The method provided by this invention refines the dynamic target processing and raster update process. By comparing raster probability differences across multiple consecutive historical images, it identifies potential dynamic rasters and forms a dynamic target candidate set, enabling rapid capture of dynamic targets. Secondly, based on Kalman filtering to predict target positions, it achieves accurate prediction of dynamic targets. By weighted probability updates and dynamically adjusting weight coefficients according to target speed, it avoids the problems of lag in fast targets and instability in slow targets caused by fixed weight coefficients. In summary, the method provided by this invention achieves rapid detection, accurate prediction, and adaptive updating of dynamic targets, enabling maps to reflect dynamic target movement in real time and reducing navigation safety risks caused by dynamic target lag.

[0112] In some embodiments, after S5 described above, the method provided by the present invention further includes: The target raster map is smoothed using a Gaussian filter algorithm, with the filter kernel size determined based on the preset raster size of the target raster map. Real-time image data of the target scene is acquired at a preset frequency using a binocular camera, and the target raster map is updated based on this real-time image data.

[0113] In one example, the method provided by this invention uses Gaussian filtering and other methods to smooth the target raster map. The Gaussian filter kernel size is set according to the raster size; for example, when the raster size is 0.1m × 0.1m, the filter kernel is set to 3×3, and the filter standard deviation is set to 0.8-1.2. Through smoothing, the problem of abrupt changes in raster occupancy probability caused by point cloud data noise and dynamic target prediction errors can be reduced, making the map occupancy probability change more smoothly and the probability transition between raster cells more natural, effectively improving map smoothness and reducing interference with subsequent navigation decisions. Based on newly acquired data (new image data (left and right images) is continuously acquired at a preset frequency of 25-30 frames / second, and new fused point cloud data is obtained after preliminary steps such as data acquisition and preprocessing, depth calculation, and semantic segmentation), the raster map is updated in a timely manner. Specifically, for each new frame of fused point cloud data acquired, the core calculation steps of target raster map construction and dynamic target processing are repeated, and the occupancy probability and semantic attributes of the corresponding raster in the map are recalculated and corrected to ensure that the map can reflect changes in the environment in real time. This enables effective control of the map update cycle, maintains the timeliness and accuracy of the map, and meets the real-time environmental perception requirements of autonomous driving and robot navigation. As described above, the method provided by this invention, through Gaussian filtering smoothing and high-frequency real-time updates, can reduce probability abrupt changes caused by point cloud noise and dynamic target prediction errors, making the probability transition between grids more natural, improving map smoothness, and avoiding interference with subsequent navigation decisions due to probability abrupt changes. Secondly, by updating the map in real-time at a preset frequency, it can ensure that the map reflects environmental changes in real time.

[0114] To facilitate understanding of this solution, the following example will be used to explain and illustrate the method provided by this invention.

[0115] In one example, the method provided by the present invention includes: data acquisition and preprocessing, deep computing and semantic segmentation, data fusion, raster map construction, dynamic target processing, and map optimization and updating.

[0116] Specifically, the grid map construction system 100 activates the binocular cameras, acquires left and right images of the scene in real time (data acquisition), performs noise reduction, correction, and enhancement on the images (preprocessing), and outputs the preprocessed images. Then, the grid map construction system 100 uses deep learning to calculate the disparity map on the preprocessed images.

[0117] The system combines camera intrinsic and extrinsic parameters to convert the data into 3D point cloud data (depth calculation). Simultaneously, the pre-processed left image is used to classify pixel semantic categories using a pre-trained semantic segmentation model (semantic segmentation), outputting 3D point cloud data and semantic information respectively. Next, the 3D point cloud data is projected onto the left image plane to match pixel positions, semantic labels are assigned based on semantic information, invalid point clouds are filtered out, and point cloud data with fused depth and semantics is output. Then, the raster map construction system 100 fuses the point cloud data, divides it into grids according to a set size, calculates the initial grid occupancy probability (combined with semantic weights), iteratively updates the occupancy probability (using Bayesian rules), assigns grid semantic attributes, and outputs the initial raster map. Following this, the raster map construction system 100 compares the initial semantic raster map with multiple consecutive frames, calculates the difference in occupancy probability, marks potential dynamic grids, clusters them to form a dynamic target candidate set (dynamic detection), establishes a six-dimensional state vector, estimates the target state using Kalman filtering (motion modeling), maps the predicted position to the map, updates the grid occupancy probability with weights (probability update), and outputs the dynamically optimized map. Finally, the dynamically optimized map is smoothed using Gaussian filtering, updated in real time with newly acquired data, and the final high-precision raster map (target raster map) is output.

[0118] In summary, the beneficial effects of the method provided by the embodiments of the present invention are as follows: This invention uses a binocular camera as the core sensor, which significantly reduces hardware costs compared to LiDAR, thereby greatly reducing system hardware investment. Simultaneously, binocular vision depth calculation effectively reduces depth error rate and improves semantic accuracy. Compared to monocular vision depth estimation, it effectively improves detection accuracy. Compared to the data sparsity problem of LiDAR, it provides richer texture and semantic information, achieving a balance between low cost and high accuracy, and lowering the application threshold for autonomous driving and robot navigation technologies. Secondly, existing technologies suffer from low accuracy in identifying small targets (bicycles, traffic cones) and high positioning errors due to insufficient utilization of semantic information. This invention effectively improves the accuracy of small target segmentation through multi-category fine semantic segmentation, enhances fusion accuracy through pixel-level semantic and deep fusion, and employs a dynamic adjustment mechanism for semantic weights based on environmental complexity and historical data feedback. These measures significantly improve the accuracy of small target identification, effectively reduce positioning errors, and substantially enhance the perception of small targets, avoiding navigation risks caused by misjudgment of small targets and improving the precision and reliability of environmental perception. Next, the method provided by this invention addresses the high latency in distinguishing dynamic obstacles and the long map update cycle in existing technologies. slow To address this issue, this invention utilizes multi-frame raster difference detection, Kalman filtering for dynamic target prediction, and flexible setting of the real-time map update cycle to achieve rapid identification, accurate prediction, and timely map updates for dynamic targets. Compared to existing technologies, the processing latency for dynamic scenes is effectively reduced, map timeliness is significantly improved, and the real-time reflection of dynamic obstacle position changes provides timely environmental data support for autonomous driving obstacle avoidance and robot path planning, reducing safety hazards caused by map lag. The method flow of this invention covers the entire process from data acquisition, processing, fusion, map generation, dynamic optimization to iterative updates, with each step logically connected and seamlessly integrated. Simultaneously, the system allows users to adjust key parameters such as grid size, filtering parameters, and weight coefficient thresholds according to application scenarios (autonomous driving, robot navigation, etc.), adapting to scenarios with varying accuracy requirements and environmental complexity. For example, an autonomous driving scenario can select a 0.1m × 0.1m × 0.1m grid size, while a robot navigation scenario can select a 0.05m × 0.05m × 0.05m grid size, meeting diverse practical application needs and enhancing the versatility and adaptability of the technical solution.

[0119] Finally, the Gaussian filtering, semi-global matching, DBSCAN clustering, and Kalman filtering algorithms used in this invention are all mature and reliable computer vision and signal processing algorithms with low technical barriers and easy engineering implementation. Furthermore, the value ranges of each key parameter (such as filter kernel size 3×3-5×5, cluster radius 0.3-0.5 meters, weight coefficient 0.3-0.8, etc.) and calculation formulas (such as occupancy probability calculation, Kalman filter prediction and update formulas) are clearly given, making the technical solution clearly operable and verifiable, facilitating subsequent engineering development and productization, and shortening the technology transformation cycle.

[0120] The foregoing mainly describes the solutions of the embodiments of the present invention from a methodological perspective. It is understood that, in order to achieve the above-mentioned functions, the grid map construction system 100 includes at least one of the hardware structures and software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present invention.

[0121] In this embodiment of the invention, the raster map construction system 100 can be divided into functional units according to the above method example. For example, the raster map construction system 100 can be divided into functional units corresponding to various functions, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0122] For example, Figure 4 This diagram illustrates the hardware structure of a raster map construction system according to an embodiment of the present invention. The raster map construction system 100 includes: an image acquisition module 110, used to acquire image data of a target scene using a binocular camera; an image processing module 120, used to perform depth calculation and semantic segmentation on the image data to obtain 3D point cloud data and semantic information corresponding to the target scene, wherein the 3D point cloud data includes a set of point clouds corresponding to the 3D spatial positions of each object in the target scene, and the semantic information is the pixel category label corresponding to each pixel in the image data; a raster processing module 130, used to divide the 3D space of the target scene into multiple rasteres according to a preset raster size; to perform fusion processing on the 3D point cloud data and semantic information to obtain the pixel category label corresponding to each point cloud in the 3D point cloud data; to determine the number of effective point clouds in each raster, wherein effective point clouds are those located within the left image range and whose depth values ​​are within the effective detection range; and to determine the occupancy probability and semantic attributes of each raster based on the number of effective point clouds and the pixel category label corresponding to each effective point cloud, wherein the semantic attribute of each raster is the pixel category label of the most prevalent effective point cloud in each raster. The map generation module 140 is used to generate an initial raster map based on the occupancy probability and semantic attributes of each raster. The map update module 150 is used to update the occupancy probability and semantic attributes of each raster in the initial raster map based on dynamic targets in the target scene, thereby obtaining the target raster map.

[0123] It should be understood that specific descriptions of the above-mentioned optional methods can be found in the foregoing method embodiments, and will not be repeated here. Furthermore, explanations of any of the raster map construction systems 100 provided above, as well as descriptions of their beneficial effects, can be found in the corresponding method embodiments described above, and will not be repeated here.

[0124] This invention also provides an electronic device including a memory and one or more processors. The memory is coupled to the processors. The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the method as described in any of the above implementations.

[0125] This invention also provides a computer-readable storage medium storing at least one computer instruction, which is loaded and executed by a processor to implement the methods of the various embodiments described above. Explanations of the relevant content and descriptions of the beneficial effects of any of the computer-readable storage media provided above can be found in the corresponding embodiments described above, and will not be repeated here.

[0126] This invention also provides a computer program product that, when run on a computer, causes the computer to execute the method in any of the above implementations.

[0127] This invention also provides a chip. This chip integrates a control circuit for implementing the functions of the aforementioned grid map construction system 100 and one or more ports. Optionally, the functions supported by this chip are as described above and will not be repeated here.

[0128] Those skilled in the art will understand that the program for implementing all or part of the steps of the above embodiments, which can be executed by a program instructing related hardware, can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The processing unit or processor mentioned above can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0129] This invention also provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform any of the methods described in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this invention is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.

[0130] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of the present invention, such as, but not limited to, the aforementioned memory, computer-readable storage medium, and communication chip, are all non-transitory. Those skilled in the art should recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0131] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for constructing a raster map based on binocular vision and semantic segmentation, characterized in that, The method includes: Image data of the target scene is acquired using a binocular camera; Depth calculation and semantic segmentation are performed on the image data respectively to obtain the three-dimensional point cloud data and semantic information corresponding to the target scene. The three-dimensional point cloud data includes a set of point clouds corresponding to the three-dimensional spatial position of each object in the target scene, and the semantic information is the pixel category label corresponding to each pixel in the image data. The three-dimensional space of the target scene is divided into multiple grids according to a preset grid size; the three-dimensional point cloud data and semantic information are fused to obtain the pixel category label corresponding to each point cloud in the three-dimensional point cloud data; the number of effective point clouds in each grid is determined, wherein the effective point clouds are those located within the left image range and whose depth values ​​are within the effective detection range; the occupancy probability and semantic attributes of each grid are determined according to the number of effective point clouds and the pixel category label corresponding to each effective point cloud, wherein the semantic attribute of each grid is the pixel category label of the effective point cloud with the highest proportion in each grid; An initial raster map is generated based on the occupancy probability and semantic attributes of each raster. The occupancy probability and semantic attributes of each grid cell in the initial grid map are updated based on the dynamic targets in the target scene to obtain the target grid map.

2. The raster map construction method based on binocular vision and semantic segmentation according to claim 1, characterized in that, Before performing depth calculation and semantic segmentation on the image data respectively, the method further includes: The image data is preprocessed, including: denoising the image data of the target scene using a Gaussian filtering algorithm; correcting radial and tangential distortion of the denoised image data based on the calibration parameters of the binocular camera; and enhancing the contrast of the corrected image data using a histogram equalization algorithm.

3. The raster map construction method based on binocular vision and semantic segmentation according to claim 2, characterized in that, The image data includes the left and right images of the target scene; depth calculation and semantic segmentation are performed on the image data, including: Determine the disparity map of the target scene based on the left and right images of the target scene; Based on the basic information of the binocular camera, the disparity map is converted into three-dimensional point cloud data. The basic information includes intrinsic parameters and extrinsic parameters. The intrinsic parameters include focal length, principal point coordinates and distortion coefficients. The extrinsic parameters include rotation matrix and translation vector. The left image of the target scene is divided into pixel categories by a pre-trained semantic segmentation model to obtain the pixel category label corresponding to each pixel.

4. The raster map construction method based on binocular vision and semantic segmentation according to claim 3, characterized in that, The occupancy probability The formula for determining it is: ； Where n is the number of valid point clouds, W i The semantic weight coefficient is the semantic weight coefficient corresponding to the i-th valid point cloud, and each pixel category label corresponds to a different semantic weight coefficient.

5. The raster map construction method based on binocular vision and semantic segmentation according to claim 4, characterized in that, The 3D point cloud data is fused with semantic information to obtain pixel category labels corresponding to each point cloud in the 3D point cloud data, including: A projection matrix is ​​established based on the principle of perspective projection and the intrinsic parameters of the binocular camera; Project each point cloud in the 3D point cloud data onto the left image plane and match the corresponding pixel position of each point cloud in the left image. The pixel category label of each point cloud is determined based on the semantic information, thus obtaining the pixel category label corresponding to each point cloud included in the 3D point cloud data.

6. The raster map construction method based on binocular vision and semantic segmentation according to claim 5, characterized in that, After determining the occupancy probability and semantic attributes of each raster based on the number of valid point clouds and the pixel category label corresponding to each valid point cloud, the method further includes: The occupancy probability of each grid cell is updated using a Bayesian update rule; The Bayesian update rule is as follows: ； in, The updated occupancy probability. To occupy a probability in history, This represents the current occupancy probability.

7. The raster map construction method based on binocular vision and semantic segmentation according to claim 6, characterized in that, The method further includes: If the vehicle density or pedestrian density in the target scene is greater than or equal to the preset vehicle density or the preset pedestrian density is greater than or equal to the preset pedestrian density, the semantic weight coefficient corresponding to the pixel category label is increased to pedestrian, bicycle and traffic cone. When the standard deviation of all point clouds in the target scene is greater than a preset standard deviation threshold, the semantic weight coefficient corresponding to each pixel category label is reduced. If the positioning deviation rate of any target in the target scene is greater than or equal to a preset deviation rate threshold, the semantic weight coefficient corresponding to the pixel category label of the target is adjusted according to the positioning deviation rate, wherein the positioning deviation rate is determined based on historical continuous multi-frame image data of the target scene.

8. The raster map construction method based on binocular vision and semantic segmentation according to claim 7, characterized in that, The occupancy probability and semantic attributes of each grid cell in the initial grid map are updated based on dynamic targets in the target scene to obtain the target grid map, including: For each of the grid cells, the occupancy probability difference value of each grid cell at adjacent time moments is determined based on historical consecutive multi-frame images of the target scene; Grids with an occupancy probability difference value greater than or equal to a preset occupancy probability difference value threshold are identified as potential dynamic grids. Cluster analysis is performed on the potential dynamic grids to obtain a dynamic target candidate set. The dynamic target candidate set includes at least one grid cluster, each grid cluster includes one or more potential dynamic grids, and each grid cluster corresponds to a dynamic target. Based on the Kalman filter algorithm, the predicted position of each dynamic target is obtained by predicting the six-dimensional state vector of each dynamic target corresponding to the historical consecutive frames of images. The predicted location corresponding to each dynamic target is mapped to each grid cell included in the initial grid map, and the updated occupancy probability and semantic attributes of each grid cell are determined. Among them, the updated occupancy probability The formula for determining it is: ； The occupancy probability corresponding to the initial grid map. The occupancy probability is determined based on the predicted location corresponding to each dynamic target; The weighting coefficient is used; and if the updated occupancy probability is greater than 1, the updated occupancy probability is set to 1, and if the updated occupancy probability is less than 0, the updated occupancy probability is set to 0. The initial grid map is updated based on the updated occupancy probability and semantic attributes of each grid cell to obtain the target grid map; Before mapping the predicted location of each dynamic target to each grid cell in the initial grid map and determining the updated occupancy probability and semantic attributes of each grid cell, the method further includes: The weighting coefficients are determined based on the motion characteristics of each dynamic target; Specifically, when the speed of the dynamic target is greater than a first speed threshold, the weighting coefficient is a first value; when the speed of the dynamic target is less than a second speed threshold, the weighting coefficient is a second value, and the first value is less than the second value; when the speed standard deviation of the dynamic target in multiple consecutive frames is greater than a speed standard deviation threshold, the weighting coefficient is determined based on the speed standard deviation and is negatively correlated with the speed standard deviation.

9. The raster map construction method based on binocular vision and semantic segmentation according to claim 8, characterized in that, After obtaining the target raster map, the method further includes: The target raster map is smoothed using a Gaussian filtering algorithm, and the kernel size of the Gaussian filtering algorithm is determined based on the preset raster size of the target raster map. Real-time image data of the target scene is acquired at a preset frequency using a binocular camera, and the target raster map is updated based on the real-time image data.

10. A raster map construction system based on binocular vision and semantic segmentation, applied to the raster map construction method based on binocular vision and semantic segmentation as described in any one of claims 1-9, characterized in that, The system includes: The image acquisition module is used to acquire image data of the target scene through a binocular camera; The image processing module is used to perform depth calculation and semantic segmentation on the image data to obtain three-dimensional point cloud data and semantic information corresponding to the target scene. The three-dimensional point cloud data includes a set of point clouds corresponding to the three-dimensional spatial position of each object in the target scene, and the semantic information is the pixel category label corresponding to each pixel in the image data. The grid processing module is used to divide the three-dimensional space of the target scene into multiple grids according to a preset grid size; to perform fusion processing on the three-dimensional point cloud data and semantic information to obtain the pixel category label corresponding to each point cloud included in the three-dimensional point cloud data; to determine the number of effective point clouds included in each grid in the multiple grids, wherein the effective point clouds are point clouds located within the left image range and whose depth values ​​are within the effective detection range; and to determine the occupancy probability and semantic attributes of each grid according to the number of effective point clouds and the pixel category label corresponding to each effective point cloud, wherein the semantic attribute of each grid is the pixel category label of the effective point cloud with the highest proportion in each grid. The map generation module is used to generate an initial raster map based on the occupancy probability and semantic attributes of each raster. The map update module is used to update the occupancy probability and semantic attributes of each grid cell in the initial grid map according to the dynamic targets in the target scene, so as to obtain the target grid map.

Images