Semantic segmentation training method and system suitable for blind glasses

Abstract

The invention discloses a semantic segmentation training method suitable for blind glasses. The semantic segmentation training method comprises three stages of vision-noise label alignment training, Bayesian reasoning and ELBO optimization and real-time reasoning. In the first stage, a preliminary relation between visual features and space-related noise labels is established, so that the model understands a space distribution rule of the noise labels; in the second stage, through variation reasoning approximate posteriori distribution, model parameters are optimized to minimize noise influence, and model robustness and segmentation accuracy are improved; in the third stage, on the basis of the optimized model, a semantic segmentation result of a clean label is rapidly generated for visual data collected by the blind glasses in real time, and accurate and timely environment information is provided for a user. According to the method, noise label interference is effectively reduced through staged training, segmentation robustness and real-time performance of the model in a complex environment are improved, reliable environment perception support is provided for blind users, and application innovation of auxiliary vision equipment in the barrier-free field is promoted.

Description

Technical Field

[0001] This invention relates to the field of eyeglasses for the blind, and more specifically to a semantic segmentation training method and system suitable for eyeglasses for the blind. Background Technology

[0002] While glasses for the blind rely on real-time scene segmentation technology for environmental perception, they face significant label noise issues. On one hand, labeling using multi-sensor fusion (such as cameras and LiDAR) is affected by device characteristics and the environment, resulting in boundary definition errors due to edge blurring and reflectivity differences. On the other hand, moving objects, lighting, and weather changes in dynamic environments cause mismatches between statically labeled tags and real-time data features; for example, differences in day and night lighting can lead to misclassification of road labels. Existing methods ignore the spatial correlation of label noise—the spatial relationships between objects in the scene (such as the adjacency of roads and curbs) are not utilized. The method of independently processing pixel or data point labels cannot correct noise through spatial context, resulting in weak anti-interference capabilities and decreased accuracy of the segmentation model, making it difficult to meet the needs of complex dynamic scenes. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a Bayesian semantic segmentation method based on spatial correlation distribution. By introducing latent Gaussian variables to model the spatial dependence of pixel-level label errors, and utilizing the Kac-Murdock-Szegö (KMS) matrix to efficiently handle high-dimensional spatial correlation, this method solves the overfitting problem of ordinary segmentation algorithms under noisy labels and improves the real-time semantic segmentation accuracy and robustness of blind glasses in complex environments.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a semantic segmentation training method suitable for glasses for the blind, characterized by comprising the following steps: Step 1: Enter the visual-noise label alignment training stage to establish a preliminary connection between visual features and labels containing spatially correlated noise, enabling the model to understand the spatial distribution pattern of noise labels. Step two: Enter the Bayesian inference and ELBO optimization stage. By approximating the posterior distribution through variational inference, the model parameters are optimized to minimize the impact of noise labels on the segmentation results, thereby improving the robustness of the model in noisy environments and the segmentation accuracy. Step three: Enter the real-time inference stage. Based on the model optimized in step two, the visual sensor input data collected in real time by the blind person's glasses is used to quickly generate semantic segmentation results with clean labels, providing blind users with accurate and timely environmental information.

[0005] As a further improvement to the present invention, the specific steps of the visual-noise label alignment training in step one are as follows: Step 1: Extract image features using a pre-trained visual encoder; Steps one and two involve modeling spatially correlated noise. Specifically, the model defines the predicted label y and the true label y. The relationship is: in, Indicates that given model parameters Given an input image x, the probability of observing label y appearing; Indicates a clean label Given the conditional probability of observing label y, we can determine it by introducing latent variables. The Gaussian distribution is used to introduce spatial correlation; Indicates that given model parameters Given the input image x, clean labels The probability of occurrence; Step 13: Construct the KMS matrix, a one-dimensional KMS matrix. Defined as: ; Where n represents the dimension of the matrix, corresponding to the number of pixels in the image; It is a parameter with a value between (-1, 1) used to control the correlation strength between label errors of adjacent pixels. The closer the value is to 1, the stronger the correlation between the label errors of adjacent pixels. Step 14: In a two-dimensional scene, use the Kronecker product. Extended, where V is a diagonal matrix, ensuring a computational complexity of O(HW), where H and W are the height and width of the image, respectively; Step 15: Using the model and data built in Steps 11 to 14, update the parameters of the visual encoder through the backpropagation algorithm.

[0006] As a further improvement to the present invention, the Bayesian inference and ELBO optimization stages in step two are as follows: Stochastic gradient descent or its variants are used to alternately optimize the model parameters and variational distribution parameters. In each iteration, the variational distribution parameters are first fixed, and the model parameters are updated using the optimization algorithm by calculating the gradient of ELBO with respect to the model parameters, so that the model can better interpret the observation data. Then, the model parameters are fixed, the gradient of ELBO with respect to the variational distribution parameters is calculated, and the variational distribution parameters are updated to make the variational distribution closer to the true posterior distribution. This process is repeated until ELBO converges.

[0007] As a further improvement to the present invention, the real-time inference stage in step three is specifically as follows: Step 31, Input Processing: The images captured in real time by the camera of the blind person's glasses are preprocessed, including grayscale conversion and histogram equalization. Then, the sliding window method is used to generate image blocks of fixed size. These image blocks are input into the optimized visual encoder to extract features. Step 32, perform latent variable inference: using the Bayesian inference module, calculate the posterior distribution of the latent variables based on the input image features and model parameters; Step 33, Generate Segmentation Results: Based on clean labels ( The posterior probability of the image is used to generate a pixel-level segmentation mask using the maximum a posteriori estimation method. The maximum a posteriori estimation selects the label with the highest posterior probability as the predicted label for each pixel, thereby obtaining the semantic segmentation result of the entire image. Finally, the segmentation mask is output to the speech or tactile feedback module and converted into information that blind users can perceive.

[0008] Another aspect of the present invention provides a system comprising: A visual encoder is used to extract image features and output pixel-level feature representations. ,in, Image size, Number of feature channels; A large language model is used as the core of semantic reasoning, receiving visual and speech features to generate environmental semantic descriptions and navigation instructions. The noise label modeling module is used to introduce latent variables. Let represent the log-odds ratio of pixel-level label errors, whose prior distribution is a Gaussian distribution. The covariance matrix adopts the KMS matrix structure to capture the spatial correlation of adjacent pixel label errors; The Bayesian inference module is used to jointly estimate model parameters and the posterior distribution of latent variables through lower bound optimization of evidence, and to derive a probabilistic model with spatial correlation.

[0009] The beneficial effects of this invention compared to existing solutions are as follows: Significantly improved noise robustness: Explicit modeling of the spatial correlation of label errors effectively addresses the mislabeling of continuous regions in blind glasses caused by changes in illumination and sensor noise. Under moderate noise levels, it improves performance by 12%-15% compared to ordinary segmentation methods, approaching the effect of clean label training.

[0010] Real-time computation efficiency: By leveraging the structured properties of the KMS matrix, the complexity of operations such as matrix inversion and determinant calculation is reduced from O(N^3) to O(N) (where N is the number of pixels), achieving millisecond-level real-time response on edge computing devices and meeting the immediacy requirements of navigation for the blind.

[0011] Accurate spatial dependency modeling: by adjusting relevant parameters (0≤ρ<1), it flexibly captures the spatial correlation of obstacle boundaries and texture change areas, improves the boundary segmentation accuracy of key obstacles such as stairs and steps by 20%, and reduces the risk of misjudgment. Detailed Implementation

[0012] The present invention will be further described in detail below with reference to the given embodiments.

[0013] This embodiment presents a semantic segmentation training method suitable for visually impaired people's glasses, which employs a multi-stage training approach, as detailed below: Visual-noise label alignment training phase Objective: To establish a preliminary connection between visual features and labels containing spatially correlated noise, enabling the model to understand the spatial distribution patterns of the noise labels.

[0014] Visual feature extraction: Extracting image features using a pre-trained visual encoder (such as a variant of Vision Transformer).

[0015] Spatial correlation noise modeling: Define the observed predicted label y (i.e., the actual labeled label with noise) and the true label The relationship (ideally accurate labeling) is as follows: ,in, Indicates that given model parameters Given an input image x, the probability of observing label y appearing; Indicates a clean label Given the conditional probability of observing label y, we can determine it by introducing latent variables. The Gaussian distribution is used to introduce spatial correlation; Indicates that given model parameters Given the input image x, clean labels The probability of occurrence.

[0016] KMS matrix construction: One-dimensional KMS matrix Defined as: Here, n represents the dimension of the matrix, corresponding to the number of pixels in the image; It is a parameter with a value between (-1, 1) used to control the correlation strength between label errors of adjacent pixels. The closer the value is to 1, the stronger the correlation between the label errors of adjacent pixels.

[0017] In a two-dimensional scene, using the Kronecker product In this extension, V is a diagonal matrix, ensuring a computational complexity of O(HW), where H and W are the height and width of the image, respectively, thus maintaining high computational efficiency even when processing large-scale image data.

[0018] Model Training: Using the model and data constructed above, the parameters of the visual encoder are updated through the backpropagation algorithm. During this process, the model continuously adjusts its parameters, attempting to learn the correspondence between visual features and noisy labels. Although the labels are noisy at this stage, the model can initially adapt to this data distribution, providing a foundation for more accurate optimization later.

[0019] Bayesian Inference and ELBO Optimization Phase Objective: To improve the robustness and segmentation accuracy of the model in noisy environments by approximating the posterior distribution through variational inference, optimizing model parameters to minimize the impact of noise labels on segmentation results.

[0020] Bayesian inference principle: Bayesian inference is a probability-based inference method that allows us to update our beliefs (i.e., prior distributions) about model parameters based on new evidence (i.e., observed data). In this invention, because labels are noisy, we cannot directly rely on observed labels for accurate semantic segmentation; therefore, we need to use Bayesian inference to infer a cleaner label distribution that is closer to reality.

[0021] The role of ELBO optimization: Evidence Lower Bound (ELBO) is an important tool for optimizing Bayesian models. It indirectly optimizes model parameters by constructing a computable lower bound, enabling the model to better fit the observed data. In the case of noisy labels, the ELBO optimization process can be understood as follows: on the one hand, it tries to match the model-generated label distribution as closely as possible to the observed labels; on the other hand, it models and corrects for label noise by introducing the distribution of latent variables.

[0022] The specific optimization process involves using stochastic gradient descent (SGD) or its variants (such as Adam) to alternately optimize the model parameters and variational distribution parameters. In each iteration, the variational distribution parameters are first fixed, and the model parameters are updated using the optimization algorithm by calculating the gradient of ELBO with respect to the model parameters, allowing the model to better interpret the observed data. Then, the model parameters are fixed again, and the gradient of ELBO with respect to the variational distribution parameters is calculated to update the variational distribution parameters, making the variational distribution closer to the true posterior distribution. This process is repeated until ELBO converges, at which point the model parameters reach their optimal state, enabling accurate inference of the clean label distribution even when considering noisy labels.

[0023] Real-time inference stage Objective: Based on the optimized model, to quickly generate clean-labeled semantic segmentation results from the real-time visual sensor input data collected by blind people's glasses, providing blind users with accurate and timely environmental information.

[0024] Input processing: The images captured in real-time by the camera on the blind person's glasses undergo preprocessing, including grayscale conversion (converting color images to grayscale to reduce data dimensionality) and histogram equalization (enhancing image contrast and making image features more prominent). Then, a sliding window method is used to generate fixed-size image patches, which are input into the optimized visual encoder to extract features. In this way, the real-time captured images can be converted into a format that the model can process, and effective feature information can be extracted.

[0025] Latent variable inference: The Bayesian inference module calculates the posterior distribution of latent variables based on the input image features and model parameters. Since the model has been optimized during the training phase, the spatial distribution of the label error can now be accurately inferred based on the input image and model parameters, i.e., the distribution of latent variables in the current image.

[0026] Segmentation result generation: based on clean labels ( The posterior probability of each pixel is used to generate a pixel-level segmentation mask using the Maximum A posteriori (MAP) estimation method. MAP selects the label with the highest posterior probability as the predicted label for each pixel, thus obtaining the semantic segmentation result for the entire image. Finally, the segmentation mask is output to a voice or tactile feedback module, converting it into information that blind users can perceive, such as voice navigation instructions (informing users of obstacles, stairs, etc.) or tactile vibration cues (using different vibration patterns to represent different environmental information), helping blind users understand their surroundings.

[0027] Through the above three stages, semantic segmentation training can be achieved simply and quickly. In this embodiment, a system is also provided for executing the program corresponding to the above method. This system includes: Visual encoder: Extracts image features and outputs pixel-level feature representations. ,in Image size, This represents the number of feature channels.

[0028] Large Language Model (LLM): As the core of semantic reasoning, it receives visual and speech features and generates environmental semantic descriptions and navigation instructions.

[0029] Noise Label Modeling Module: Introducing Latent Variables The log-odds ratio representing pixel-level label error has a prior distribution that is Gaussian. The covariance matrix adopts the KMS matrix structure to capture the spatial correlation of label errors between adjacent pixels.

[0030] Bayesian Inference Module: This module achieves joint estimation of model parameters and the posterior distribution of latent variables through Evidence Lower Bound (ELBO) optimization, deriving a probabilistic model with spatial correlation.

[0031] In summary, this embodiment proposes a Bayesian semantic segmentation method based on spatial correlation distribution. By introducing a latent Gaussian variable to model the spatial dependence of pixel-level label errors, and utilizing the Kac-Murdock-Szegö (KMS) matrix to efficiently handle high-dimensional spatial correlation, it solves the overfitting problem of ordinary segmentation algorithms under noisy labels and improves the real-time semantic segmentation accuracy and robustness of blind glasses in complex environments.

[0032] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A semantic segmentation training method suitable for visually impaired people's glasses, characterized in that: Includes the following steps: Step 1: Enter the visual-noise label alignment training stage to establish a preliminary connection between visual features and labels containing spatially correlated noise, enabling the model to understand the spatial distribution pattern of noise labels. Step two: Enter the Bayesian inference and ELBO optimization stage. By approximating the posterior distribution through variational inference, the model parameters are optimized to minimize the impact of noise labels on the segmentation results, thereby improving the robustness of the model in noisy environments and the segmentation accuracy. Step three: Enter the real-time inference stage. Based on the model optimized in step two, the visual sensor input data collected in real time by the blind person's glasses is used to quickly generate semantic segmentation results with clean labels, providing blind users with accurate and timely environmental information.

2. The semantic segmentation training method for visually impaired glasses according to claim 1, characterized in that: The specific steps of the visual-noise label alignment training in step one are as follows: Step 1: Extract image features using a pre-trained visual encoder; Steps one and two involve modeling spatially correlated noise. Specifically, the model defines the predicted label y and the true label y. The relationship is: ； in, Indicates that given model parameters Given an input image x, the probability of observing label y appearing; Indicates a clean label Given the conditional probability of observing label y, we can determine it by introducing latent variables. The Gaussian distribution is used to introduce spatial correlation; Indicates that given model parameters Given the input image x, clean labels The probability of occurrence; Step 13: Construct the KMS matrix, a one-dimensional KMS matrix. Defined as: ； Where n represents the dimension of the matrix, corresponding to the number of pixels in the image; It is a parameter with a value between (-1, 1) used to control the correlation strength between label errors of adjacent pixels. The closer the value is to 1, the stronger the correlation between the label errors of adjacent pixels. Step 14: In a two-dimensional scene, use the Kronecker product. Extended, where V is a diagonal matrix, ensuring a computational complexity of O(HW), where H and W are the height and width of the image, respectively; Step 15: Using the model and data built in Steps 11 to 14, update the parameters of the visual encoder through the backpropagation algorithm.

3. The semantic segmentation training method for visually impaired glasses according to claim 1 or 2, characterized in that: The Bayesian inference and ELBO optimization phases in step two are as follows: Stochastic gradient descent or its variants are used to alternately optimize the model parameters and variational distribution parameters. In each iteration, the variational distribution parameters are first fixed, and the model parameters are updated using the optimization algorithm by calculating the gradient of ELBO with respect to the model parameters, so that the model can better interpret the observation data. Then, the model parameters are fixed, the gradient of ELBO with respect to the variational distribution parameters is calculated, and the variational distribution parameters are updated to make the variational distribution closer to the true posterior distribution. This process is repeated until ELBO converges.

4. The semantic segmentation training method for visually impaired glasses according to claim 1 or 2, characterized in that: The real-time inference stage in step three is as follows: Step 31, Input Processing: The images captured in real time by the camera of the blind person's glasses are preprocessed, including grayscale conversion and histogram equalization. Then, the sliding window method is used to generate image blocks of fixed size. These image blocks are input into the optimized visual encoder to extract features. Step 32, perform latent variable inference: using the Bayesian inference module, calculate the posterior distribution of latent variables based on the input image features and model parameters; Step 33, Generate Segmentation Results: Based on clean labels ( The posterior probability of the image is used to generate a pixel-level segmentation mask using the maximum a posteriori estimation method. The maximum a posteriori estimation selects the label with the highest posterior probability as the predicted label for each pixel, thereby obtaining the semantic segmentation result of the entire image. Finally, the segmentation mask is output to the speech or tactile feedback module and converted into information that blind users can perceive.

5. A semantic segmentation system suitable for eyeglasses for the blind, used to execute the semantic segmentation training method for eyeglasses for the blind described in any one of steps 1 to 4, characterized in that: include: A visual encoder is used to extract image features and output pixel-level feature representations. ,in, Image size, Number of feature channels; A large language model is used as the core of semantic reasoning, receiving visual and speech features to generate environmental semantic descriptions and navigation instructions. The noise label modeling module is used to introduce latent variables. Let represent the log-odds ratio of pixel-level label errors, whose prior distribution is a Gaussian distribution. The covariance matrix adopts the KMS matrix structure to capture the spatial correlation of adjacent pixel label errors; The Bayesian inference module is used to jointly estimate model parameters and the posterior distribution of latent variables through lower bound optimization of evidence, and to derive a probabilistic model with spatial correlation.

Images