A stereoscopic perception scene-based image semantic segmentation method and system

Abstract

The application discloses a kind of based on stereoscopic perception scene's image semantic segmentation method and system, the method includes: obtaining parallax map, the parallax map includes left parallax map, intermediate parallax map and right parallax map;Introduce semantic uncertainty neural body rendering module, construct scene stereoscopic perception semantic segmentation network model;Based on scene stereoscopic perception semantic segmentation network model, parallax map is handled to semantic segmentation identification, and rendered semantic segmentation parallax map is obtained.The system includes: acquisition module, construction module and segmentation module.The application can understand the stereoscopic perception network model of scene from spatial dimension by introducing semantic uncertainty neural body rendering module, and realizes fine scene semantic segmentation.The application is a kind of based on stereoscopic perception scene's image semantic segmentation method and system, and can be widely applied in semantic image segmentation technical field.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation technology, and in particular to an image semantic segmentation method and system based on stereo perception scenes. Background Technology

[0002] Scene semantic understanding, due to its ability to capture the semantic relationships between objects within a scene, has seen widespread application in artificial intelligence in recent years, covering fields such as autonomous driving, robotic services and autonomous decision-making, intelligent monitoring and security, and AR / VR. Despite the significant success of semantic segmentation in scene semantic understanding, achieving refined scene understanding in complex environments remains challenging. In the real world, different objects possess varying depth and positional relationships. These objects may overlap or partially occlude, making it difficult to accurately distinguish their boundaries and contours during segmentation. Therefore, effectively utilizing stereo information is crucial for achieving refined scene understanding. Existing technologies employ depth map fusion to effectively utilize stereo information for understanding the world. However, most methods attempt to construct an architecture that complements and fuses RGB and depth features to improve the network's stereo perception capability. In this process, the model focuses on combining color and depth features through weighted summation, stitching, attention, or gating mechanisms, without truly understanding the scene from a spatial dimension. This makes it difficult to accurately capture the spatial relationships between features, limiting the model's ability to understand the true three-dimensional appearance of the scene. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide an image semantic segmentation method and system based on stereo perception scenes. By introducing a semantic uncertainty neural body rendering module, the stereo perception network model of the scene can be understood from the spatial dimension, thereby achieving fine scene semantic segmentation.

[0004] The first technical solution adopted in this invention is: an image semantic segmentation method based on stereo perception scene, comprising the following steps:

[0005] Obtain a disparity map, which includes a left disparity map, a middle disparity map, and a right disparity map;

[0006] A semantic uncertainty neural body rendering module is introduced to construct a scene stereo perception semantic segmentation network model;

[0007] A scene stereo perception semantic segmentation network model is used to perform semantic segmentation and recognition processing on the disparity map to obtain a rendered semantic segmentation disparity map.

[0008] Furthermore, the scene stereo perception semantic segmentation network model includes a volume feature construction module, a dimension separation feature module, a neural radiation field optimization feature prediction module, and a semantic uncertainty neural body rendering module.

[0009] Furthermore, the step of performing semantic segmentation and recognition processing on the disparity map based on the scene stereo perception semantic segmentation network model to obtain the rendered semantic segmentation disparity map specifically includes:

[0010] Input the disparity map into a scene-based stereo perception semantic segmentation network model;

[0011] A volume feature construction module based on a scene stereo perception semantic segmentation network model is used to extract volume features from the disparity map to obtain the volume features of the disparity map.

[0012] The dimension separation feature module based on the scene stereo perception semantic segmentation network model performs linear interpolation sampling on the volume features of the disparity map to obtain the geometric structure features of the disparity map.

[0013] The neural radiation field optimization feature prediction module based on the scene stereo perception semantic segmentation network model optimizes the geometric structure features of the disparity map to obtain the optimized geometric structure features.

[0014] The semantic uncertainty neural body rendering module based on the scene stereo perception semantic segmentation network model renders the optimized geometric structure features to obtain the rendered semantic segmentation disparity map.

[0015] Furthermore, the volume feature construction module based on the scene stereo perception semantic segmentation network model performs volume feature extraction processing on the disparity map to obtain the volume features of the disparity map. This step specifically includes:

[0016] The disparity map is input into the volume feature construction module of the scene stereo perception semantic segmentation network model, which includes the Res34Unet network and the 3D Unet network.

[0017] The Res34Unet network based on the volume feature construction module extracts and processes the disparity map to obtain the 2D image features of the disparity map. The 2D image features of the disparity map include the 2D image features of the left disparity map, the 2D image features of the middle disparity map, and the 2D image features of the right disparity map.

[0018] The variances of the 2D image features of the middle disparity map, the left disparity map, and the right disparity map are obtained respectively, and the cost volume is constructed.

[0019] The 3D Unet network based on the volume feature construction module extracts volume features from the cost volume to obtain the volume features of the disparity map.

[0020] Furthermore, the step of the dimension separation feature module based on the scene stereo perception semantic segmentation network model performing linear interpolation sampling on the volume features of the disparity map to obtain the geometric structure features of the disparity map specifically includes:

[0021] The volumetric features of the disparity map are input into the dimension separation feature module of the scene stereo perception semantic segmentation network model. The dimension separation feature module includes a neural radiation field and an interpolation sampling module.

[0022] Based on the neural radiation field of the dimension separation feature module, ray sampling processing is performed on the intermediate disparity map in the disparity map to obtain the depth map of the intermediate disparity map.

[0023] The interpolation sampling module based on the dimension separation feature module performs trilinear interpolation on the depth map and volume features of the disparity map to obtain the geometric structure features of the disparity map.

[0024] Furthermore, the neural radiation field optimization feature prediction module based on the scene stereo perception semantic segmentation network model optimizes the geometric structure features of the disparity map to obtain optimized geometric structure features. This step specifically includes:

[0025] The disparity map is preprocessed to obtain the preprocessed disparity map;

[0026] The preprocessed disparity map is stitched together with the geometric structural features of the disparity map to obtain the stitched geometric structural features.

[0027] Based on the first multilayer MLP network in the neural radiation field optimization feature prediction module, the spliced ​​geometric structure features are optimized to obtain semantic features and color density features.

[0028] Given the location of any sampling point and the unit direction vector of the line of sight, the color density features are processed by projection sampling and bilinear interpolation to obtain the pixel color and geometric structure features of the color density features.

[0029] The sampling point position, the viewing unit direction vector, the color density feature, the pixel color of the color density feature, and the geometric structure feature of the color density feature are concatenated to obtain the concatenated color density feature.

[0030] Based on the second multilayer MLP network in the neural radiation field optimization feature prediction module, the spliced ​​color density features are optimized to obtain the density and color corresponding to the sampling point location.

[0031] By integrating semantic features, color density features, and the density and color corresponding to the sampling point locations, the optimized geometric structure features are obtained.

[0032] Furthermore, the step of preprocessing the disparity map to obtain a preprocessed disparity map specifically includes:

[0033] The depth map of the intermediate disparity map is projected onto the left disparity map, the intermediate disparity map, and the right disparity map respectively, and bilinear interpolation is performed to extract the pixel color and pixel 2D features.

[0034] Obtain the mean and variance of the 2D features of the pixels;

[0035] By integrating pixel color, pixel 2D features, the mean of pixel 2D features, and the variance of pixel 2D features, a preprocessed disparity map is obtained.

[0036] Furthermore, the semantic uncertainty neural body rendering module based on the scene stereo perception semantic segmentation network model renders the optimized geometric structure features to obtain the rendered semantic segmentation disparity map. This step specifically includes:

[0037] Determine the transparency of the optimized geometric features;

[0038] Based on transparency, the semantic uncertainty neural body rendering module renders the optimized geometric structure features and combines the density and color corresponding to the sampling point positions to obtain the pixel semantic distribution.

[0039] Calculate the softmax distribution of the pixel semantic distribution to obtain the multi-class probability of the pixel;

[0040] Render the pixels of the depth map in the intermediate disparity map to obtain the pixel colors;

[0041] By integrating the multi-class probabilities of pixels and the color of pixels, a semantic segmentation disparity map is obtained after rendering.

[0042] Furthermore, the step of determining the transparency of the optimized geometric structural features specifically includes:

[0043] The semantic distribution of the predicted sampling points is calculated using softmax to obtain the multi-class probabilities.

[0044] The semantic distribution uncertainty of the optimized geometric structure features is determined based on multi-class probabilities;

[0045] The transparency of the optimized geometric features is determined based on the semantic distribution uncertainty.

[0046] The second technical solution adopted in this invention is: an image semantic segmentation system based on stereo perception scene, comprising:

[0047] The acquisition module is used to acquire disparity maps;

[0048] The module is used to introduce the semantic uncertainty neural body rendering module and build a scene stereo perception semantic segmentation network model.

[0049] The segmentation module is used to perform semantic segmentation and recognition processing on the disparity map based on the scene stereo perception semantic segmentation network model, and obtain the rendered semantic segmentation disparity map.

[0050] The beneficial effects of the method and system of this invention are as follows: This invention acquires a disparity map and further constructs a scene stereo perception semantic segmentation network model. Based on this model, semantic segmentation and recognition processing is performed on the disparity map. The model input consists of three disparity maps and a depth map from the perspective of the intermediate disparity map. The model output is the semantic segmentation map of the intermediate disparity map. The scene stereo perception semantic segmentation network model understands the scene from a spatial dimension, thus more accurately identifying the boundaries and contours of scene elements during scene segmentation, improving the precision of the segmentation results. In the semantic uncertainty neural body rendering module within the scene stereo perception semantic segmentation network model, when rendering the semantics and color of pixels at each sampling point of the neural radiation field, the transparency of each sampling point of the ray is calculated using a hybrid semantic uncertainty approach. Semantic uncertainty neural body rendering allows the network to continuously balance the estimated density and semantic uncertainty at sampling points during training, improving the performance of semantic segmentation by utilizing the consistency constraint between the two. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the steps of an image semantic segmentation method based on a stereoscopic perception scene according to an embodiment of the present invention.

[0052] Figure 2 This is a structural block diagram of an image semantic segmentation system based on a stereo perception scene according to an embodiment of the present invention;

[0053] Figure 3 This is a structural block diagram of a scene stereo perception semantic segmentation network model according to a specific embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the Res34Unet network structure in the volume feature construction module of a specific embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram of the 3dUnet network in the volume feature construction module of a specific embodiment of the present invention. Detailed Implementation

[0056] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0057] Reference Figure 1 This invention provides an image semantic segmentation method based on stereo perception scenes, the method comprising the following steps:

[0058] S1. Obtain a disparity map, which includes a left disparity map, a middle disparity map, and a right disparity map;

[0059] Specifically, the left parallax map, the middle parallax map, and the right parallax map were captured simultaneously by three cameras positioned at the same height on the left, center, and right.

[0060] S2. Introduce a semantic uncertainty neural body rendering module to construct a scene stereo perception semantic segmentation network model;

[0061] Specifically, a network model for achieving fine semantic segmentation based on scene stereo perception is proposed, such as... Figure 3 As shown, the model input consists of three disparity maps and a depth map from the perspective of the intermediate disparity map. The model output is a semantic segmentation map of the intermediate disparity map. The network model includes: a volume feature construction module, a dimension separation feature module, a neural radiation field optimization feature prediction module, and a semantic uncertainty neural body rendering module.

[0062] S3. Based on the scene stereo perception semantic segmentation network model, the disparity map is semantically segmented and recognized to obtain the rendered semantic segmentation disparity map.

[0063] S31, Volume Feature Construction Module;

[0064] Specifically, firstly, this invention uses a shared Res34Unet network, such as Figure 4 As shown, dense 2D image features are extracted from three disparity maps. Then, using the camera parameters of each of the three disparity maps, the 2D image features of the middle disparity map are projected onto 128 scanned stereo planes. The variance between these features and the features projected onto these planes from the left and right disparity maps is calculated to construct the cost volume. Subsequently, the cost volume is input into the 3D Unet network, as shown... Figure 5 As shown, the output is the volumetric feature of channel 32. The volumetric feature can be regarded as an explicit representation of the 3D scene structure from the perspective of the intermediate disparity map. It describes the meaningful scene geometry and contour information from this perspective.

[0065] S32, Dimensional Separation Feature Module;

[0066] Specifically, the feature separation module based on depth maps is achieved through ray sampling using neural radiation fields based on depth maps. First, a ray is emitted from the camera's optical center at the intermediate disparity map viewpoint, passing through each pixel, and 32 points are sampled at equal intervals within a pixel depth of ±0.5 meters. For each sampled point, geometric structure features are obtained by trilinear interpolation from volumetric features. These geometric structure features describe the spatial structure of the sampled point locations from the intermediate disparity map viewpoint. Ray sampling using neural radiation fields based on depth maps provides an intuitive way to spatially separate features of elements at different depths in a scene. Subsequently, the semantics of each separated element feature are understood, achieving the goal of stereoscopic perception of the scene from a spatial perspective.

[0067] S33, Neural Radiation Field Optimization Feature Prediction Module;

[0068] Specifically, since the representation of the 3D scene structure at the sampling point by geometric structural features is relatively coarse and viewpoint-limited, we use the unique new viewpoint synthesis characteristics and high-quality 3D reconstruction capabilities of neural radiation fields to optimize the geometric structural features. The optimized geometric structural features better represent the spatial structure outline and details of the sampling point location, helping the network to perceive and understand the environment more accurately and make high-quality semantic segmentation performance.

[0069] S331. Multi-view feature aggregation optimizes geometric structure features to achieve semantic prediction of sampling points;

[0070] Specifically, in this invention, the unit direction vector of the line of sight under the perspective of the intermediate disparity map is d∈R. 3 The position of each sampling point of the ray x∈R 3 The pixel colors are extracted by projecting them onto three disparity maps and using bilinear interpolation. and pixel 2D features

[0071] Furthermore, compared to three-dimensional points in free space, three-dimensional points on a curved surface are more likely to have a consistent local appearance across multiple views; therefore, this invention also simultaneously calculates... The mean u and variance v of the sampling points will be used to determine the geometric structural features f. v By splicing together the above features, we obtain Subsequently, a multi-layer MLP network A (MLP) A ) Optimize f c The optimized output is the semantic feature f at sampling point position x. s and color density features f cd Here, f c It contains the geometric structural features f of the space where the sampling point x is located. v Image color C observed from multiple perspectives i and feature f iAnd the consistent local appearance u, v of multi-view image features, thus enabling MLP A The scene structure is understood prior to the semantic segmentation process. Additionally, multi-layer MLPs... A In predicting semantic features f s It also predicts the color density feature f. cd Color density feature f cd Subsequently, the implicit representations of scene density and color are passed into the neural radiation field, thus semantic features f s It also indirectly incorporates scene geometric information.

[0072] Finally, since observing a single point of a ray in isolation and making a prediction would lack contextual information about the ray, this invention uses the semantic feature sequence (f) of N sampling points of the ray. s (x1), ..., f s (x N A 4-head, single-layer transformer self-attention module is used to predict the semantic distribution s of N sampling points. The dimension of the semantic distribution s is equal to the number of categories.

[0073] S332, Prediction of sampling point color and density;

[0074] Specifically, given the location x of any sampling point of the ray and the unit direction vector d of the ray, the color density feature f is obtained. cd Then, the pixel color is extracted by bilinear interpolation by projecting the sampling point position x onto three disparity maps. and geometric structural features f v splicing, splicing features Passing into another multi-level MLP branch B (MLP) B Predict the density σ and color c at sampling point x. It should be noted that the input MLP... B The first x-axis undergoes a NDC spatial coordinate transformation projected onto the intermediate disparity map viewpoint. Specifically, the NDC coordinates consist of three elements: the normalized pixel coordinates projected from the sampling point x onto the intermediate disparity map viewpoint pixel plane, and the normalized depth. Additionally, the input MLP... B The coordinates of the gaze unit direction vector d are also transformed by projecting onto the intermediate disparity map viewpoint. Using a normalized NDC space allows training the neural radiation field to focus on learning the density, color, and semantic predictions of a pixel in the intermediate disparity map at different depths, rather than relying on variable-scale world coordinates, thus controlling the problem scale of scene construction.

[0075] S34, Semantic Uncertainty Neural Rendering Module.

[0076] Specifically, the semantic distribution s predicted for the k-th sampling point on the ray. k Calculate the softmax function to obtain the multi-class probability ρk The expression for multi-class probability is:

[0077] ρ k =siftmax(s k )

[0078] In the above formula, ρ k Represents multi-class probability, softmax(·) denotes the softmax function, s k This represents the semantic distribution at the k-th sampling point of the ray predicted by the model;

[0079] Using ρ k Calculate the semantic distribution uncertainty δ at the k-th sampling point of the ray. k Its expression is:

[0080]

[0081] In the above formula, δ k denoted by , where numClass represents the semantic distribution uncertainty at the k-th sampling point of the ray;

[0082] Mixed δ k Calculate the transparency a at the k-th sampling point of the ray. k The expression for transparency is:

[0083]

[0084] In the above formula, a k The transparency at the k-th sampling point of the ray is represented by t, t represents the current training epoch, T represents the total training epoch, and σ is the total transparency at the k-th sampling point of the ray. k This represents the density of the ray at the k-th sampling point predicted by the model;

[0085] This invention employs a regularization strategy. In the early stages of training, semantic uncertainty has a significant impact on the calculation of transparency. As training progresses, the impact of semantic uncertainty on the calculation of transparency is gradually reduced.

[0086] Next, the semantics of each pixel are rendered from the perspective of the intermediate disparity map. First, a ray is emitted from the camera optical center to each pixel, and the color c, density σ, and semantics s of N sampling points on the ray are queried. The pixel semantic distribution is calculated by accumulating the radiance of all sampling points on this ray. The expression for pixel semantic distribution is:

[0087]

[0088] in;

[0089]

[0090] In the above formula, Represents the pixel semantic distribution, where N represents the number of ray sampling points, k represents the index of the sampling point on the ray, and s k Let a represent the semantic distribution at the k-th sampling point predicted by the model. k a j These represent the transparency at the k-th and j-th sampling points on the ray, respectively.

[0091] Finally, a softmax function is calculated to obtain the multi-class probability of pixels based on their semantic distribution. The expression for the multi-class probability of a pixel is:

[0092]

[0093] In the above formula, Represents the multi-class probability of a pixel;

[0094] In addition, we will render the color of each pixel from the intermediate disparity map viewpoint, and similarly, we will calculate the pixel color by accumulating the shading points upsampled by the ray. Its expression is:

[0095]

[0096] In the above formula, c represents the color of a pixel. k This represents the color at the k-th sampling point of the ray predicted by the model.

[0097] Reference Figure 2 An image semantic segmentation system based on stereo perception scenes, comprising:

[0098] The acquisition module is used to acquire disparity maps;

[0099] The module is used to introduce the semantic uncertainty neural body rendering module and build a scene stereo perception semantic segmentation network model.

[0100] The segmentation module is used to perform semantic segmentation and recognition processing on the disparity map based on the scene stereo perception semantic segmentation network model, and obtain the rendered semantic segmentation disparity map.

[0101] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0102] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An image semantic segmentation method based on stereo perception scenes, characterized in that, Includes the following steps: Obtain a disparity map, which includes a left disparity map, a middle disparity map, and a right disparity map; A semantic uncertainty neural body rendering module is introduced to construct a scene stereo perception semantic segmentation network model; Input the disparity map into a scene-based stereo perception semantic segmentation network model; A volume feature construction module based on a scene stereo perception semantic segmentation network model is used to extract volume features from the disparity map to obtain the volume features of the disparity map. The dimension separation feature module based on the scene stereo perception semantic segmentation network model performs linear interpolation sampling on the volume features of the disparity map to obtain the geometric structure features of the disparity map. The neural radiation field optimization feature prediction module based on the scene stereo perception semantic segmentation network model optimizes the geometric structure features of the disparity map to obtain the optimized geometric structure features. The semantic uncertainty neural body rendering module, based on a scene stereo perception semantic segmentation network model, renders the optimized geometric structure features to obtain a rendered semantic segmentation disparity map, including: The semantic distribution of the predicted sampling points is calculated using softmax to obtain the multi-class probabilities. The expression for the multi-class probability is: ； In the above formula, Represents multi-class probability, This represents the softmax function. The model predicts the ray number 1. Semantic distribution at each sampling point; The semantic distribution uncertainty of the optimized geometric structure features is determined based on multi-class probabilities; The expression for the semantic distribution uncertainty is: ； In the above formula, Indicates the first ray Semantic distribution uncertainty at each sampling point Indicates the number of semantic categories; The transparency of the optimized geometric structure features is determined based on the semantic distribution uncertainty; The expression for transparency is: ； In the above formula, Indicates the first ray Transparency at each sampling point This indicates the current training epoch. Indicates the total training epoch. The model predicts the ray number 1. Density at each sampling point; Based on transparency, the semantic uncertainty neural body rendering module renders the optimized geometric structure features and combines the density and color corresponding to the sampling point positions to obtain the pixel semantic distribution. First, emit rays from the camera's optical center to each pixel, and then query the ray... The color of each sampling point ,density and semantics The pixel semantic distribution is calculated by accumulating the radiance of all sampling points along this ray. The expression for pixel semantic distribution is: ； in: ； In the above formula, Represents the semantic distribution of pixels. Indicates the number of ray sampling points. Indicates the index of the sampling point on the ray. The model predicts the first The semantic distribution at each sampling point , They represent the first ray on the ray. The and the first Transparency at each sampling point; Calculate the softmax distribution of the pixel semantic distribution to obtain the multi-class probability of the pixel; The expression for the multi-class probability of the pixel is: ； In the above formula, Represents the multi-class probability of a pixel; Render the pixels of the depth map in the intermediate disparity map to obtain the pixel colors; The expression for the pixel color is: ； In the above formula, Represents the color of a pixel. The model predicts the ray number 1. The color at each sampling point; By integrating the multi-class probabilities of pixels and the color of pixels, a semantic segmentation disparity map is obtained after rendering.

2. The image semantic segmentation method based on stereo perception scene according to claim 1, characterized in that, The scene stereo perception semantic segmentation network model includes a volume feature construction module, a dimension separation feature module, a neural radiation field optimization feature prediction module, and a semantic uncertainty neural body rendering module.

3. The image semantic segmentation method based on stereo perception scene according to claim 2, characterized in that, The volume feature construction module based on the scene stereo perception semantic segmentation network model performs volume feature extraction processing on the disparity map to obtain the volume features of the disparity map. This step specifically includes: The disparity map is input into the volume feature construction module of the scene stereo perception semantic segmentation network model, which includes the Res34Unet network and the 3D Unet network. The Res34Unet network based on the volume feature construction module extracts and processes the disparity map to obtain the 2D image features of the disparity map. The 2D image features of the disparity map include the 2D image features of the left disparity map, the 2D image features of the middle disparity map, and the 2D image features of the right disparity map. The variances of the 2D image features of the middle disparity map, the left disparity map, and the right disparity map are obtained respectively, and the cost volume is constructed. The 3D Unet network based on the volume feature construction module extracts volume features from the cost volume to obtain the volume features of the disparity map.

4. The image semantic segmentation method based on stereo perception scene according to claim 3, characterized in that, The dimension separation feature module of the scene stereo perception semantic segmentation network model performs linear interpolation sampling on the volume features of the disparity map to obtain the geometric structure features of the disparity map. This step specifically includes: The volumetric features of the disparity map are input into the dimension separation feature module of the scene stereo perception semantic segmentation network model. The dimension separation feature module includes a neural radiation field and an interpolation sampling module. Based on the neural radiation field of the dimension separation feature module, ray sampling processing is performed on the intermediate disparity map in the disparity map to obtain the depth map of the intermediate disparity map. The interpolation sampling module based on the dimension separation feature module performs trilinear interpolation on the depth map and volume features of the disparity map to obtain the geometric structure features of the disparity map.

5. The image semantic segmentation method based on stereo perception scene according to claim 4, characterized in that, The neural radiation field optimization feature prediction module based on the scene stereo perception semantic segmentation network model optimizes the geometric structure features of the disparity map to obtain optimized geometric structure features. This step specifically includes: The disparity map is preprocessed to obtain the preprocessed disparity map; The preprocessed disparity map is stitched together with the geometric structural features of the disparity map to obtain the stitched geometric structural features. Based on the first multilayer MLP network in the neural radiation field optimization feature prediction module, the spliced ​​geometric structure features are optimized to obtain semantic features and color density features. Given the location of any sampling point and the unit direction vector of the line of sight, the color density features are processed by projection sampling and bilinear interpolation to obtain the pixel color and geometric structure features of the color density features. The sampling point position, the viewing unit direction vector, the color density feature, the pixel color of the color density feature, and the geometric structure feature of the color density feature are concatenated to obtain the concatenated color density feature. Based on the second multilayer MLP network in the neural radiation field optimization feature prediction module, the spliced ​​color density features are optimized to obtain the density and color corresponding to the sampling point location. By integrating semantic features, color density features, and the density and color corresponding to the sampling point locations, the optimized geometric structure features are obtained.

6. The image semantic segmentation method based on stereo perception scene according to claim 5, characterized in that, The step of preprocessing the disparity map to obtain a preprocessed disparity map specifically includes: The depth map of the intermediate disparity map is projected onto the left disparity map, the intermediate disparity map, and the right disparity map respectively, and bilinear interpolation is performed to extract the pixel color and pixel 2D features. Obtain the mean and variance of the 2D features of the pixels; By integrating pixel color, pixel 2D features, the mean of pixel 2D features, and the variance of pixel 2D features, a preprocessed disparity map is obtained.

7. An image semantic segmentation system based on stereo perception scenes, characterized in that, Includes the following modules: The acquisition module is used to acquire disparity maps; The module is used to introduce the semantic uncertainty neural body rendering module and build a scene stereo perception semantic segmentation network model. The segmentation module is used to input the disparity map into the scene-based stereo perception semantic segmentation network model; A volume feature construction module based on a scene stereo perception semantic segmentation network model is used to extract volume features from the disparity map to obtain the volume features of the disparity map. The dimension separation feature module based on the scene stereo perception semantic segmentation network model performs linear interpolation sampling on the volume features of the disparity map to obtain the geometric structure features of the disparity map. The neural radiation field optimization feature prediction module based on the scene stereo perception semantic segmentation network model optimizes the geometric structure features of the disparity map to obtain the optimized geometric structure features. The semantic uncertainty neural body rendering module, based on a scene stereo perception semantic segmentation network model, renders the optimized geometric structure features to obtain a rendered semantic segmentation disparity map, including: The semantic distribution of the predicted sampling points is calculated using softmax to obtain the multi-class probabilities. The expression for the multi-class probability is: ； In the above formula, Represents multi-class probability, This represents the softmax function. The model predicts the ray number 1. Semantic distribution at each sampling point; The semantic distribution uncertainty of the optimized geometric structure features is determined based on multi-class probabilities; The expression for the semantic distribution uncertainty is: ； In the above formula, Indicates the first ray Semantic distribution uncertainty at each sampling point Indicates the number of semantic categories; The transparency of the optimized geometric structure features is determined based on the semantic distribution uncertainty; The expression for transparency is: ； In the above formula, Indicates the first ray Transparency at each sampling point This indicates the current training epoch. Indicates the total training epoch. The model predicts the ray number 1. Density at each sampling point; Based on transparency, the semantic uncertainty neural body rendering module renders the optimized geometric structure features and combines the density and color corresponding to the sampling point positions to obtain the pixel semantic distribution. First, emit rays from the camera's optical center to each pixel, and then query the ray... The color of each sampling point ,density and semantics The pixel semantic distribution is calculated by accumulating the radiance of all sampling points along this ray. The expression for pixel semantic distribution is: ； in: ； In the above formula, Represents the semantic distribution of pixels. Indicates the number of ray sampling points. Indicates the index of the sampling point on the ray. The model predicts the first The semantic distribution at each sampling point , They represent the first ray on the ray. The and the first Transparency at each sampling point; Calculate the softmax distribution of the pixel semantic distribution to obtain the multi-class probability of the pixel; The expression for the multi-class probability of the pixel is: ； In the above formula, Represents the multi-class probability of a pixel; Render the pixels of the depth map in the intermediate disparity map to obtain the pixel colors; The expression for the pixel color is: ； In the above formula, Represents the color of a pixel. The model predicts the ray number 1. The color at each sampling point; By integrating the multi-class probabilities of pixels and the color of pixels, a semantic segmentation disparity map is obtained after rendering.

Images