Lightweight real-time semantic segmentation visual recognition method and system

Abstract

This invention discloses a lightweight real-time semantic segmentation and visual recognition method and system. It acquires the image to be processed and inputs it into a dual-stream heterogeneous coding network. The network includes a high-resolution spatial detail branch and a semantic context branch that performs step-downsampling. A dynamic multi-scale sparse attention module is embedded in the semantic context branch to generate a spatial location importance score map and filter key anchor points. Global attention is performed only on key anchor points, while depthwise convolution is performed on non-key anchor points. The output features from the two branches are input into an adaptive gated feature fusion module to generate spatial adaptive fusion weights, which are then weighted and subjected to edge enhancement processing. The fused features are upsampled by a decoder to output pixel-level semantic segmentation results. This invention effectively reduces computational complexity, improves edge segmentation accuracy, and meets the real-time inference requirements of embedded devices.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, and relates to a lightweight real-time semantic segmentation and visual recognition method and system. Background Technology

[0002] Semantic segmentation, which aims to assign a category label to each pixel in an image, is a core technology for applications such as autonomous driving, robot navigation, and smart healthcare. Existing high-precision semantic segmentation models typically employ deep convolutional neural network structures, coupled with complex dilated spatial pyramid pooling modules to extract multi-scale features. While these models can achieve high segmentation accuracy, they have a huge number of parameters and high computational latency, making it difficult to implement real-time inference on resource-constrained embedded devices such as automotive chips or drone flight control boards.

[0003] To address the aforementioned issues, existing technologies have proposed various lightweight solutions. One approach replaces the original heavy network structure with a lightweight backbone network. Another approach utilizes model compression techniques to prune or quantize the trained model. Yet another approach designs dedicated lightweight architectures, such as employing a dual-branch structure to extract spatial detail information and semantic context information separately. However, these existing solutions still have significant drawbacks. First, existing methods typically use fixed weighting methods or simple element-wise addition when fusing features from different levels. This static fusion strategy cannot dynamically adjust the receptive field and the importance of feature channels based on the complexity of the input image. This leads to computational redundancy in simple scenarios and a tendency to lose crucial details in complex scenarios.

[0004] Secondly, existing attention mechanisms face the problem of excessively high computational costs when introduced into lightweight models. Traditional self-attention mechanisms require calculating the correlation between all pixel pairs, and their computational complexity increases quadratically with image resolution. Even when introduced into lightweight models, this significantly increases inference latency, undermining real-time performance advantages. While existing linear attention or window attention mechanisms reduce computation to some extent, they still fall short in maintaining global context modeling capabilities and struggle to capture long-distance dependencies. Furthermore, existing techniques generally suffer from information distortion during decoder upsampling. Traditional bilinear interpolation or transposed convolution upsampling methods are fixed and cannot adaptively recover high-frequency edge details lost during downsampling. This directly leads to blurred segmentation boundaries and low recognition accuracy for small objects.

[0005] Therefore, there is an urgent need for a new technology that can significantly reduce model complexity and computational costs while ensuring high accuracy, in order to meet the pressing needs of real-time performance and embedded deployment in practical applications. Summary of the Invention

[0006] To address the problems existing in the background technology, this invention proposes a lightweight real-time semantic segmentation and visual recognition method and system.

[0007] The first aspect of this application provides a lightweight real-time semantic segmentation and visual recognition method, including: The image to be processed is acquired and input into a dual-stream heterogeneous coding network, which includes a spatial detail branch that maintains high resolution and a semantic context branch that performs step-downsampling. Dynamic multi-scale sparse attention modules are embedded at different stages of the semantic context branch. The dynamic multi-scale sparse attention modules are used to generate spatial location importance scoring maps and filter key anchor locations. Global attention calculation is performed only on the key anchor locations, and depth convolution operation is performed on non-key anchor locations. The output features of the spatial detail branch and the semantic context branch are input into the adaptive gated feature fusion module. The adaptive gated feature fusion module is used to generate spatial adaptive fusion weights and perform weighted fusion and edge enhancement processing on the two features. The fused features are upsampled by the decoder to output pixel-level semantic segmentation results.

[0008] Optionally, the process of generating a spatial location importance score map by the dynamic multi-scale sparse attention module includes: performing a global average pooling operation on the input feature map to obtain a global feature vector; passing the global feature vector sequentially through a first convolutional layer, a nonlinear activation function, and a second convolutional layer to obtain a scalar score; and mapping the scalar score to the spatial location importance score map through a sigmoid activation function.

[0009] Optionally, the process of the dynamic multi-scale sparse attention module to select key anchor point locations includes: sorting all pixel values ​​in the spatial location importance scoring map; selecting a preset number of pixel locations with the largest sorted values ​​as key anchor point locations; generating a binary mask with the same size as the input feature map; setting the mask value corresponding to the key anchor point location as the first value; and setting the mask values ​​corresponding to the other locations as the second value.

[0010] Optionally, the process of the dynamic multi-scale sparse attention module performing global attention calculation on key anchor locations includes: projecting the input feature map through the query weight matrix, key weight matrix, and value weight matrix to obtain the query matrix, key matrix, and value matrix, respectively; for each key anchor location, calculating the dot product of the query vector at that location and the key vectors at all locations, dividing by the scaling factor, and then performing a softmax operation to obtain the attention weights; and weighted summing the attention weights with the value vectors at the corresponding locations to obtain the attention output features of that key anchor location.

[0011] Optionally, the process of the dynamic multi-scale sparse attention module performing depthwise convolution operations on non-critical anchor positions includes: identifying all non-critical anchor positions whose values ​​in the binary mask are the second value; directly processing the input features corresponding to the non-critical anchor positions through a depthwise separable convolutional layer to obtain the output features of the non-critical regions; combining the attention output features of the critical anchor positions with the output features of the non-critical regions and obtaining the final output of the module through a channel recovery convolutional layer.

[0012] Optionally, the process of generating spatial adaptive fusion weights by the adaptive gated feature fusion module includes: concatenating deep semantic features from the semantic context branch with shallow spatial features from the spatial detail branch in the channel dimension; extracting spatial features from the concatenated features through a deep convolutional layer; and mapping the output of the deep convolutional layer through a sigmoid activation function to obtain a first fusion weight matrix and a second fusion weight matrix.

[0013] Optionally, the process of weighted fusion and edge enhancement processing performed by the adaptive gated feature fusion module includes: performing edge detection operation on the shallow spatial features using the Sobel operator to obtain an edge feature map; mapping the edge feature map to the same number of channels through a point convolutional layer to obtain an edge enhancement term; multiplying the deep semantic features element-wise with the first fusion weight matrix, multiplying the shallow spatial features element-wise with the second fusion weight matrix, and adding the result of the multiplication to the edge enhancement term multiplied by a learnable scalar parameter to obtain the fused features.

[0014] Optionally, the training process of the method employs a composite loss function, which consists of a cross-entropy loss term, a Dice loss term, and a boundary-aware loss term. The calculation process of the boundary-aware loss term includes: performing morphological gradient operations on the predicted segmentation result and the true label respectively to obtain a predicted boundary map and a true boundary map; and calculating the mean square error between the predicted boundary map and the true boundary map as the boundary-aware loss value.

[0015] A second aspect of this application provides a lightweight real-time semantic segmentation and visual recognition system, comprising: The image acquisition module is used to acquire the image to be processed; The processor module is used to input the image to be processed into the dual-stream heterogeneous coding network, use the dynamic multi-scale sparse attention module to perform sparse attention calculation on the semantic context branch features, use the adaptive gated feature fusion module to perform weighted fusion and edge enhancement processing on the spatial detail branch features and semantic context branch features, and output pixel-level semantic segmentation results. The output module is used to display or transmit the pixel-level semantic segmentation results.

[0016] Optionally, the processor module includes an embedded graphics processor or a neural network acceleration unit; the embedded graphics processor or neural network acceleration unit is configured to perform depthwise separable convolution operations of the spatial detail branch and sparse matrix multiplication operations of the semantic context branch in parallel; the system further includes a storage module for storing pre-trained weight parameters and intermediate feature map data.

[0017] Compared with the prior art, the present invention has the following beneficial effects: First, by employing a dual-stream heterogeneous coding network structure, the requirements for spatial detail preservation and semantic context extraction are effectively balanced. The spatial detail branch maintains high-resolution feature transmission, avoiding the loss of edge information caused by traditional downsampling operations. The semantic context branch performs step-wise downsampling, ensuring the full aggregation of deep semantic information. This architecture design significantly reduces the overall computational complexity while maintaining segmentation accuracy.

[0018] Secondly, the introduction of a dynamic multi-scale sparse attention module significantly improves computational efficiency. This module intelligently selects key anchor points by generating a spatial importance scoring map. Global attention computation is performed only on key regions, while depthwise convolution operations are used on non-key regions. This differentiated processing strategy avoids the redundant computational overhead of full-pixel computation in traditional self-attention mechanisms. While maintaining the ability to model global context, the computational complexity is reduced from quadratic to linear, thus meeting the real-time inference requirements of embedded devices.

[0019] Furthermore, the adaptive gating feature fusion module achieves dynamic optimization and integration of features. This module can automatically generate spatially adaptive fusion weights based on the input image content, replacing the traditional fixed weighting method. Combined with edge enhancement processing using the Sobel operator, high-frequency details lost during upsampling are effectively recovered. This makes the final pixel-level semantic segmentation results clearer and sharper at object boundaries, significantly improving the recognition accuracy of small target objects.

[0020] Finally, a composite loss function consisting of cross-entropy loss, set overlap loss, and boundary-aware loss is used for training to further enhance the model's robustness. The boundary-aware loss term specifically constrains the segmentation edges, prompting the model to pay more attention to contour accuracy. The overall technical solution achieves an optimal balance between parameter count, computational latency, and segmentation accuracy, making it particularly suitable for resource-constrained applications and applications with extremely high real-time requirements, such as autonomous driving and robot navigation. Attached Figure Description

[0021] Figure 1 This is a flowchart of a lightweight real-time semantic segmentation and visual recognition method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a lightweight real-time semantic segmentation and visual recognition system according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In one embodiment, such as Figure 1 As shown, a lightweight real-time semantic segmentation and visual recognition method is provided, which is then applied to... Figure 1 Taking China as an example, the following specific steps will be used: S10: Acquire the image to be processed and input it into a dual-stream heterogeneous coding network, wherein the dual-stream heterogeneous coding network includes a spatial detail branch that maintains high resolution and a semantic context branch that performs step-downsampling.

[0024] Specifically, the process of acquiring the image to be processed and inputting it into the two-stream heterogeneous coding network is as follows: First, the image acquisition module acquires the original color image and adjusts it to a preset resolution, such as 512 pixels by 512 pixels. Then, the adjusted image is simultaneously fed into two independent branches of the two-stream heterogeneous coding network.

[0025] The spatial detail branch aims to preserve high-frequency edge information in the image. This branch consists of multiple cascaded depthwise separable convolutional blocks. Throughout the feature extraction process, the spatial detail branch maintains the spatial resolution of the feature map. Specifically, after the input image passes through the first convolutional layer, the feature map size remains 512 pixels by 512 pixels. Subsequent convolutional operations do not employ downsampling strategies with a stride greater than 1, nor do they use pooling layers to reduce the size. This design ensures that shallow texture details and object contours are not blurred or lost during transmission.

[0026] The semantic context branch aims to capture global semantic information and multi-scale contextual features. This branch employs a stepped downsampling architecture comprising multiple stages. In the first stage, the input image undergoes convolution and a stride of 2, halving the feature map size to 256 pixels by 256 pixels. In the second stage, the features are downsampled again to 128 pixels by 128 pixels. As network depth increases, the feature map size decreases progressively while the number of channels increases progressively. This stepped structure allows the network to achieve a larger receptive field at deeper layers, thereby effectively understanding scene categories and spatial relationships between objects in the image.

[0027] The two branches are structurally independent but functionally complementary. The spatial detail branch focuses on the layer-by-layer propagation of high-resolution features, avoiding the detail annihilation problem caused by multiple downsampling in traditional encoders. The semantic context branch aggregates global information by progressively reducing resolution, solving the problem that a single high-resolution network cannot capture long-range dependencies. The dual-stream heterogeneous design enables the network to process visual information of different granularities in parallel, providing rich and high-quality input sources for subsequent feature fusion.

[0028] This invention's high-resolution spatial detail branch effectively enhances the recognition capability of small target objects, preventing the loss of subtle structures during downsampling. The semantic context branch, employing stepped downsampling, achieves thorough extraction of deep semantics while controlling computational load. The combination of these two approaches enables the encoding network to output feature representations with both fine spatial location and rich semantic content, even with a relatively low parameter count, laying a solid foundation for real-time, high-precision semantic segmentation.

[0029] S20: Embed dynamic multi-scale sparse attention modules at different stages of the semantic context branch. The dynamic multi-scale sparse attention modules are used to generate spatial location importance scoring maps and filter key anchor points. Global attention calculation is performed only on the key anchor points, and depth convolution operations are performed on non-key anchor points.

[0030] Specifically, a dynamic multi-scale sparse attention module is embedded into multiple downsampling stages of the semantic context branch. This module first receives feature maps from the previous layer as input. The input feature maps then enter the scoring generation sub-network. The scoring generation sub-network consists of lightweight convolutional layers and activation functions to quickly predict the importance score for each spatial location. Through normalization, these scores are transformed into spatial location importance scoring maps. Each value in the scoring map represents the probability that the corresponding pixel location contains key semantic information.

[0031] Subsequently, the module filters key anchor points based on the spatial importance score map. Specifically, the system sets a dynamic threshold or selects the top 20% of high-scoring locations as key anchor points. The remaining unselected locations are marked as non-key anchor points. This filtering mechanism allows the model to adaptively focus on object edges, corners, and complex textured areas in the image, while ignoring redundant areas such as the background.

[0032] For the selected key anchor points, the module performs global attention computation. The system extracts the query vector for each key anchor point and interacts with key-value pairs across the entire graph. This process establishes long-distance dependencies, enabling local details to perceive global context information. For non-key anchor points, the module performs depthwise convolution operations. Depthwise convolution only performs weighted summation within the local neighborhood, avoiding the high computational cost of the entire graph. This differentiated processing strategy significantly reduces the overall computational load.

[0033] In terms of multi-scale processing, this module employs different receptive field settings at different stages. In the shallow stages of the semantic context branch, the module focuses on capturing fine-grained local structures, where key anchors are densely distributed. In the deep stages, the module focuses on aggregating a wide range of semantic concepts, where key anchors are sparser but cover a broader area. Through this multi-scale embedding approach, the network can effectively balance detail preservation and semantic abstraction at different levels.

[0034] This invention generates a spatial importance score map and filters key anchor points, allowing computational resources to be concentrated on the regions with the most information. Global attention is performed only on key anchor points, overcoming the bottleneck of computational complexity increasing quadratically with resolution in traditional self-attention mechanisms. Depthwise convolution operations are performed on non-key anchor points, significantly reducing floating-point operations while maintaining feature continuity. This dynamic sparsity mechanism maintains high-precision global modeling capabilities while significantly improving inference speed, enabling the model to run in real-time on resource-constrained embedded devices. It effectively solves the technical challenge of existing lightweight models struggling to balance global context and computational efficiency.

[0035] S30: Input the output features of the spatial detail branch and the semantic context branch into the adaptive gated feature fusion module. The adaptive gated feature fusion module is used to generate spatial adaptive fusion weights and perform weighted fusion and edge enhancement processing on the two features.

[0036] Specifically, the adaptive gated feature fusion module receives a high-resolution feature map from the spatial detail branch and a deep semantic feature map from the semantic context branch. First, the module performs channel alignment on the two input features to ensure they have the same channel dimension. Then, the aligned features are concatenated and input into a gated weight generation network. This network, composed of convolutional layers and a sigmoid activation function, dynamically predicts the fusion coefficient for each pixel based on the local content of the input image. The generated output is the spatially adaptive fusion weight map, which is spatially non-uniform and intelligently indicates whether more emphasis should be placed on detail information or semantic information in different regions.

[0037] Next, the module utilizes spatial adaptive fusion weights to perform weighted fusion of the two feature paths. Specifically, the feature map of the spatial detail branch is multiplied by the weight map, while the feature map of the semantic context branch is multiplied by the complement of the weight map, and then the two are added together. This mechanism ensures that in object boundaries and regions with complex textures, the fusion result retains more high-frequency information from the spatial detail branch. However, within the object or in regions with flat backgrounds, the fusion result relies more on the global semantic information from the semantic context branch. In this way, adaptive selection and complementarity of feature content are achieved.

[0038] After weighted fusion, the module further performs edge enhancement processing. This process introduces the Sobel operator to calculate the gradient of the fused features, extracting potential edge response maps. Subsequently, the edge response maps are superimposed back into the fused features through a lightweight residual connection. This process explicitly enhances the high-frequency components in the feature maps, compensating for any loss of contour sharpness during upsampling or convolution. The feature maps after edge enhancement not only retain rich semantic category information but also possess clear and accurate descriptions of object boundaries.

[0039] This invention generates spatially adaptive fusion weights to replace traditional fixed-ratio fusion or simple addition methods, solving the problem of inconsistent detail and semantic requirements in different image regions. Weighted fusion and edge enhancement of the two feature streams effectively eliminate misalignment during multi-scale feature alignment, significantly improving the localization accuracy of segmentation boundaries. This fusion strategy enables the final output feature representation to possess both the powerful semantic discrimination capabilities of deep networks and the fine spatial localization capabilities of shallow networks. Especially when dealing with small objects and complex boundary scenes, this method significantly reduces false positives and false negatives, providing a solid guarantee for the subsequent decoder to output high-quality pixel-level semantic segmentation results, while not introducing excessive additional computational latency, maintaining the system's real-time performance.

[0040] S40: The fused features are upsampled by the decoder to output pixel-level semantic segmentation results.

[0041] Specifically, the decoder receives the fused feature map processed by the adaptive gated feature fusion module. This fused feature map typically has a low spatial resolution but contains rich semantic information. The decoder first gradually restores the spatial size of the feature map through multi-level upsampling operations. Each level of upsampling uses a bilinear interpolation algorithm or a transposed convolutional layer to expand the length and width of the feature map to a preset multiple. For example, if the input feature map size is one-quarter of the original image, it will be restored to the resolution of the original image after two levels of upsampling.

[0042] During the upsampling process, the decoder also introduces a skip connection structure. This structure directly passes high-resolution features from the corresponding level in the spatial detail branch to the corresponding stage of the decoder. The incoming high-resolution features are then added or concatenated element-wise with the upsampled features. This design effectively compensates for the shallow texture details lost during the upsampling process, enabling the recovered feature map to maintain semantic consistency while possessing a fine spatial structure.

[0043] After multiple levels of upsampling and feature refinement, the decoder outputs a feature map with the exact same size as the input image. This feature map then enters the final classification layer. The classification layer consists of a convolution operation with a kernel size of 1 pixel multiplied by 1 pixel, used to map the feature vector of each pixel to a predefined dimension of the number of categories. Subsequently, the softmax function is used to normalize the scores of each category for each pixel, generating a probability distribution map. The system determines the final category label for each pixel based on the maximum probability principle, thus outputting pixel-level semantic segmentation results.

[0044] This invention achieves accurate reconstruction from abstract semantic features to specific pixel labels by upsampling the decoder to output pixel-level semantic segmentation results. Multi-level upsampling combined with a skip connection mechanism effectively solves the problems of edge blurring and detail loss caused by continuous upsampling in traditional decoders. The final output segmentation result is sharper and clearer at object boundaries, and the semantic annotation of internal regions is more uniform and accurate. This invention ensures that the segmented map is strictly aligned with the original image in spatial location, enabling accurate identification of small objects and irregular shapes in complex scenes, significantly improving the overall perception capability and application reliability of visual recognition systems.

[0045] In one embodiment, step S20, where the dynamic multi-scale sparse attention module generates a spatial location importance scoring map, further includes the following steps: S21: Perform global average pooling on the input feature map to obtain the global feature vector.

[0046] S22: The global feature vector is passed sequentially through the first convolutional layer, the nonlinear activation function, and the second convolutional layer to obtain a scalar score.

[0047] S22: Map the scalar score to the spatial location importance score map using the sigmoid activation function.

[0048] Specifically, the process of generating the spatial importance score map involves three consecutive steps. First, the module performs a global average pooling operation on the input feature map. This operation iterates through each channel of the feature map, calculating the arithmetic mean of all spatial location values ​​within that channel. Through this processing, the feature map, which originally had height and width dimensions, is compressed into a single global feature vector. This global feature vector condenses the global semantic context information of the entire image, eliminates spatial location differences, and retains only the statistical features of the channel dimension.

[0049] Subsequently, the system sequentially passes the global feature vector through a first convolutional layer, a non-linear activation function, and a second convolutional layer to obtain a scalar score. The first convolutional layer is used to perform dimensionality reduction or expansion transformation on the global feature vector, extracting the dependencies between channels. Next, the non-linear activation function introduces non-linearity, enhancing the model's ability to express complex features. Finally, the second convolutional layer maps the processed features to a single numerical value, the scalar score. This scalar score represents the confidence or strength that the current input feature map contains key information, serving as a global adjustment factor for the subsequent generation of the weight map.

[0050] Finally, the module maps the scalar score to the spatial importance score map using a sigmoid activation function. The sigmoid activation function compresses the numerical range of the scalar score to between zero and one. Although the input is a single scalar, in the specific architecture of this embodiment, the scalar value is broadcast and expanded to the same spatial dimension as the original input feature map, or the scalar participates as a global bias term in generating a spatially varying weight distribution. The spatial importance score map generated after mapping represents the probability that the location is considered a critical region for each pixel. The closer the value is to one, the more important the location; the closer the value is to zero, the more likely the location is to be background or redundant.

[0051] This invention performs global average pooling on the input feature map to obtain a global feature vector, effectively aggregating the semantic information of the entire image and avoiding interference from local noise in importance judgment. The global feature vector is then passed sequentially through a first convolutional layer, a non-linear activation function, and a second convolutional layer to obtain a scalar score, constructing an efficient channel attention mechanism capable of adaptively learning the global contribution of features from different channels. The scalar score is mapped to the spatial location importance score map using a sigmoid activation function, ensuring that the output weight values ​​have clear probabilistic meaning and are numerically stable. This generation method has extremely low computational overhead, requiring only a small number of convolutional operations to complete the importance assessment of the entire image, providing a fast and accurate basis for subsequent selection of key anchor point locations, and significantly improving the response speed and resource allocation efficiency of the dynamic multi-scale sparse attention module.

[0052] It should be noted that the process of the dynamic multi-scale sparse attention module in selecting key anchor point locations includes: sorting all pixel values ​​in the spatial location importance scoring map; selecting a preset number of pixel locations with the largest sorted values ​​as key anchor point locations; generating a binary mask with the same size as the input feature map; setting the mask value corresponding to the key anchor point location as the first value; and setting the mask values ​​corresponding to the other locations as the second value.

[0053] Specifically, the process of filtering key anchor point locations using the dynamic multi-scale sparse attention module includes three steps: sorting, selection, and mask generation. First, the system sorts all pixel values ​​in the spatial importance scoring map. This sorting operation rearranges the values ​​of the height multiplied by the width of the scoring map in descending order, forming an ordered sequence. Through this process, the system can clearly distinguish which spatial locations have higher semantic importance and which locations belong to low-value background regions.

[0054] Subsequently, the system selects a preset number of pixel locations with the largest sorted values ​​as key anchor points. This preset number is an integer constant pre-defined based on computational resource budget and model accuracy requirements. The system directly extracts this number of elements from the top of the ordered sequence and records their two-dimensional coordinates in the original scoring image. These selected coordinate points constitute the set of key anchor points. These locations typically correspond to the edges, corners, or textured areas of objects in the image, and are core regions carrying crucial semantic information.

[0055] Next, the system generates a binary mask with the same size as the input feature map. This binary mask is a matrix with the same height and width as the input feature map. The system sets the mask value corresponding to the key anchor point position as a first value, which is usually set to 1. Simultaneously, the system sets the mask value corresponding to the remaining positions as a second value, which is usually set to 0. After this step, the positions with a value of 1 in the binary mask accurately mark the key regions that require global attention calculation, while the positions with a value of 0 mark the non-key regions that only require depthwise convolution operations.

[0056] This invention sorts all pixel values ​​in the spatial importance scoring map, ensuring the determinism and fairness of the selection process and avoiding the uncertainty caused by random sampling. A predetermined number of pixel positions with the largest sorted values ​​are selected as key anchor points, achieving strict control and quantitative allocation of computational resources. This ensures that the model always concentrates expensive global attention computations on the few positions with the most information. A binary mask of the same size as the input feature map is generated. The mask value corresponding to the key anchor point position is set as the first value, and the mask values ​​corresponding to the other positions are set as the second value, providing clear logical guidance for subsequent differential calculations. This sorting-based hard threshold selection mechanism not only significantly reduces the computational complexity of the attention mechanism from quadratic to linear, but also ensures the complete preservation of key features. It effectively solves the problem of key information loss or excessive redundant computation caused by the difficulty in setting thresholds in traditional sparse attention methods, significantly improving the inference efficiency and feature extraction quality of the model in complex scenarios.

[0057] In one embodiment, the process of the dynamic multi-scale sparse attention module performing global attention calculation on key anchor locations includes: projecting the input feature map through a query weight matrix, a key weight matrix, and a value weight matrix to obtain a query matrix, a key matrix, and a value matrix, respectively; for each key anchor location, calculating the dot product of the query vector at that location and the key vectors at all locations, dividing by a scaling factor, and then performing a softmax operation to obtain attention weights; and weighted summing the attention weights with the value vectors at the corresponding locations to obtain the attention output features of that key anchor location.

[0058] Specifically, the process of performing global attention computation at key anchor point locations by the dynamic multi-scale sparse attention module first involves feature projection. The system linearly projects the input feature map through a query weight matrix, a key weight matrix, and a value weight matrix, respectively. The query weight matrix maps the input features to a query matrix, which represents the information features that the current pixel seeks to focus on. The key weight matrix maps the input features to a key matrix, which represents the identity features that can be retrieved for all pixels. The value weight matrix maps the input features to a value matrix, which contains the actual content information of all pixels. These three matrices have the same depth in the channel dimension, ensuring compatibility with subsequent vector operations.

[0059] Next, the system calculates the correlation between each key anchor point and all locations in the entire image. Specifically, for each selected key anchor point, the system extracts its corresponding query vector in the query matrix. Then, it performs a dot product operation on this query vector with the key vectors of all spatial locations in the key matrix. The dot product result reflects the semantic similarity between the query location and any location in the image. To maintain numerical stability, the system divides the dot product result by a scaling factor. This scaling factor is typically equal to the square root of the key vector dimension. The scaled value is then input into a softmax operation for normalization. The softmax operation converts the similarity score into a probability distribution, generating attention weights. These attention weights represent the proportion of contribution each location in the image should make when synthesizing the features of the current key anchor point.

[0060] Finally, the system aggregates global information using the generated attention weights. The system performs a weighted summation of the attention weights and the value vectors at the corresponding locations to obtain the attention output features at that key anchor point. This process essentially fuses the contextual information of the entire image to the current key anchor point based on their relevance. Only locations highly semantically relevant to the key anchor point receive higher weights, thus significantly impacting the output features. Irrelevant background regions have weights close to zero, effectively suppressing their interference with the final result. The resulting attention output features exist only at the key anchor points, forming a sparse but globally semantically rich feature representation.

[0061] This invention projects the input feature map through query weight matrices, key weight matrices, and value weight matrices, enabling flexible transformation of the feature space and enhancing the model's ability to capture long-distance dependencies. For each key anchor point, the dot product of the query vector at that location and the key vectors at all locations is calculated, divided by a scaling factor, and then a softmax operation is performed. This ensures the numerical stability of the attention score and prevents gradient vanishing or exploding problems. The attention weights are weighted and summed with the corresponding value vectors to achieve a true global receptive field, allowing key anchor points to directly acquire contextual information from any region of the image. This strategy of performing global attention calculations only at key locations retains the powerful modeling capabilities of the Transformer architecture while avoiding the enormous computational cost of traditional self-attention mechanisms that calculate for all pixel pairs. This method significantly reduces memory usage and computational latency, enabling the model to run efficiently on high-resolution images while ensuring accurate modeling of object boundaries and detailed features, thus improving the overall accuracy of semantic segmentation.

[0062] In one embodiment, the process of the dynamic multi-scale sparse attention module performing depthwise convolution operations on non-critical anchor positions includes: identifying all non-critical anchor positions whose values ​​in the binary mask are second values; directly processing the input features corresponding to the non-critical anchor positions through a depthwise separable convolutional layer to obtain non-critical region output features; combining the attention output features of the critical anchor positions with the non-critical region output features and obtaining the module's final output through a channel recovery convolutional layer.

[0063] Specifically, the process of the dynamic multi-scale sparse attention module performing depthwise convolution operations on non-critical anchor point locations begins with location recognition. The system reads the binary mask generated in the previous step and scans all pixels in the mask. The system identifies all non-critical anchor point locations in the binary mask whose values ​​are the second value. These locations were determined to be regions of low semantic importance in the previous screening process, typically corresponding to the background or smooth regions with simple textures in the image. The system records the coordinate indices of these locations for subsequent separate processing.

[0064] Subsequently, the system extracts features from these identified non-critical anchor points. The system directly processes the input features corresponding to these non-critical anchor points through a depthwise separable convolutional layer to obtain the output features for the non-critical regions. The depthwise separable convolutional layer comprises two sub-steps: depthwise convolution and pointwise convolution. Depthwise convolution independently filters each channel in the spatial dimension, effectively capturing local spatial features. Pointwise convolution performs linear combination in the channel dimension, achieving information fusion between channels. Since non-critical regions typically do not require global context information, this lightweight convolutional operation is sufficient to extract their necessary local details, while significantly reducing the number of computational parameters and floating-point operations.

[0065] Finally, the system performs feature recombination and restoration. The system combines the attention output features at the key anchor points with the output features from the non-key regions. Specifically, guided by a binary mask, the system fills the corresponding coordinates of the output feature map with the attention output features rich in global semantics at the key locations, while simultaneously filling the remaining coordinates with the local convolutional output features at the non-key locations. These two features are spatially complementary, together forming a complete intermediate feature map. This intermediate feature map is then passed through a channel recovery convolutional layer to obtain the module's final output. The channel recovery convolutional layer adjusts the number of channels in the feature map to match the input dimension required by the next layer of the network, and completes the final feature smoothing and integration.

[0066] This invention identifies all non-critical anchor point locations whose median value in the binary mask is the second value, achieving accurate differentiation of regions of varying importance and laying the foundation for differentiated processing. The input features corresponding to these non-critical anchor point locations are directly processed through a depthwise separable convolutional layer. Leveraging the efficiency of depthwise separable convolution significantly reduces the processing cost of background regions, avoiding the waste of expensive global attention computational resources in these low-information areas. The attention output features of the critical anchor point locations are combined with the output features of the non-critical regions and processed through a channel recovery convolutional layer, achieving an organic fusion of global semantic information and local detail information. This hybrid computational strategy ensures both the recognition accuracy of key object regions and maintains the overall network speed, effectively resolving the contradiction between limited computational resources and model performance requirements in high-resolution image processing, and significantly improving the system's real-time response capability and energy efficiency.

[0067] In one embodiment, step S30, where the adaptive gating feature fusion module generates spatial adaptive fusion weights, further includes the following steps: S31: Concatenate the deep semantic features from the semantic context branch with the shallow spatial features from the spatial detail branch in the channel dimension; S32: Spatial feature extraction is performed on the concatenated features through a deep convolutional layer; the output of the deep convolutional layer is mapped through a sigmoid activation function to obtain the first fusion weight matrix and the second fusion weight matrix.

[0068] Specifically, the process of generating spatial adaptive fusion weights by the adaptive gating feature fusion module first involves the integration of multi-source features. The system concatenates deep semantic features from the semantic context branch with shallow spatial features from the spatial detail branch along the channel dimension. Deep semantic features are rich in high-level category information but have low spatial resolution, while shallow spatial features retain rich edge texture details but have weak semantic abstraction capabilities. Through the channel-dimensional concatenation operation, the system constructs a composite feature map containing dual information. This composite feature map maintains alignment in the spatial dimension and simultaneously accommodates data from both semantic and detail modalities in the channel dimension, providing a complete data foundation for subsequent generation of a fusion strategy that takes both into account.

[0069] Subsequently, the system performs spatial dependency modeling on the stitched composite feature map. The system extracts spatial features from the stitched features using deep convolutional layers. Each deep convolutional layer operates independently on each channel, using spatial convolution kernels that slide along the height and width directions to capture pixel relationships within the local neighborhood. This operation can perceive the spatial distribution of object boundaries, texture variations, and semantic abrupt changes in the image. The output of the deep convolutional layer reflects the differences in the degree of dependence on semantic and detail information at different spatial locations. For example, in object edge regions, the output features strongly respond to changes in spatial details; while in smooth regions inside the object, the output features tend to express stable semantic attributes.

[0070] Finally, the system transforms the extracted spatial features into specific fusion control signals. The system maps the output of the deep convolutional layer using a sigmoid activation function to obtain a first fusion weight matrix and a second fusion weight matrix. The sigmoid activation function compresses the values ​​to the range of zero to one, giving them a physical meaning of probability or proportion. In the first fusion weight matrix, the value of each pixel represents the proportion of deep semantic features adopted at the corresponding location in the final fusion result. In the second fusion weight matrix, the value of each pixel represents the proportion of shallow spatial features adopted at the corresponding location. These two matrices are spatially complementary; that is, the sum of the first and second fusion weight values ​​at the same location is always equal to one. This mapping mechanism ensures that the model can adaptively adjust the fusion strategy according to the image content, achieving pixel-level fine-grained control.

[0071] This invention concatenates deep semantic features from the semantic context branch with shallow spatial features from the spatial detail branch along the channel dimension, breaking the fixed pattern of feature fusion in traditional methods and achieving explicit interaction of multi-scale information. The concatenated features are then used for spatial feature extraction via deep convolutional layers, enabling the generation of fusion weights to no longer rely on global statistics but to sensitively perceive local spatial structure changes, enhancing the model's adaptability to complex scenes. The output of the deep convolutional layers is mapped using a sigmoid activation function to obtain the first and second fusion weight matrices, constructing a soft gating mechanism. This mechanism avoids boundary artifacts caused by hard thresholding segmentation, achieving a smooth transition from semantic-driven to detail-driven approaches. This spatially adaptive fusion method effectively solves the problem of inaccurate target localization caused by spatial information loss in deep networks, while suppressing semantic ambiguity caused by noise interference in shallow networks, significantly improving the discriminative power and robustness of feature representations, and providing high-quality feature input for subsequent accurate segmentation or detection tasks.

[0072] It should be noted that the process of weighted fusion and edge enhancement processing of the adaptive gated feature fusion module includes: performing edge detection operation on the shallow spatial features using the Sobel operator to obtain an edge feature map; mapping the edge feature map to the same number of channels through a point convolutional layer to obtain an edge enhancement term; multiplying the deep semantic features element-wise with the first fusion weight matrix, multiplying the shallow spatial features element-wise with the second fusion weight matrix, and adding the result of the multiplication to the edge enhancement term multiplied by a learnable scalar parameter to obtain the fused feature.

[0073] Specifically, the adaptive gated feature fusion module's weighted fusion and edge enhancement process begins with the explicit extraction of edge information. The system uses the Sobel operator to perform edge detection on the shallow spatial features to obtain an edge feature map. The Sobel operator contains two convolutional kernels, one horizontal and one vertical, which can sensitively capture regions in the image where grayscale values ​​change drastically. By calculating the gradient magnitude, the system generates an edge feature map that highlights object contours and texture boundaries. This edge feature map clearly marks the structural boundaries in the scene, providing clear guidance for subsequent detail enhancement.

[0074] Subsequently, the system maps the extracted edge information to the same dimensional space as the main feature flow. The system then maps the edge feature map to the same number of channels using a point convolutional layer to obtain edge enhancement terms. The point convolutional layer performs a linear transformation on the channels of the edge feature map without changing the spatial resolution, ensuring that the number of channels matches the number of channels in the deep semantic features and shallow spatial features. This operation ensures that the edge enhancement terms can be directly added to the main feature map mathematically, transforming simple geometric boundary information into feature vectors rich in channel semantics, allowing them to be effectively integrated into the overall feature representation.

[0075] Next, the system performs weighted multi-source feature fusion. The system multiplies the deep semantic features element-wise with the first fusion weight matrix to obtain weighted semantic components. Simultaneously, the system multiplies the shallow spatial features element-wise with the second fusion weight matrix to obtain weighted detail components. The first and second fusion weight matrices were previously dynamically generated based on the importance of spatial location, controlling the contribution ratio of semantic and detail information in different regions, respectively. Through element-wise multiplication, the system preserves key semantics while suppressing background noise and enhancing spatial response in regions requiring detail.

[0076] Finally, the system integrates all components to generate the final fused feature. The system multiplies the result by the edge enhancement term multiplied by a learnable scalar parameter and then adds the result to obtain the fused feature. The learnable scalar parameter is a coefficient automatically optimized during network training to adjust the contribution of the edge enhancement term to the final result. The system first multiplies the edge enhancement term by this scalar parameter to control the injection of edge information and prevent over-enhancement that could lead to artifacts. Subsequently, the weighted semantic components, weighted detail components, and scaled edge enhancement term are added pixel-by-pixel. This process achieves an organic unity of semantic context, spatial details, and explicit edge structure, outputting a fused feature that possesses both high-level semantic discriminative power and fine-grained boundary localization capabilities.

[0077] This invention utilizes the Sobel operator to perform edge detection on the shallow spatial features, introducing prior knowledge from traditional image processing to overcome the shortcomings of purely data-driven methods in capturing minute boundaries, significantly improving the model's sensitivity to object contours. By mapping the edge feature map to the same number of channels through point convolutional layers, seamless integration of edge information with the deep feature space is achieved, enabling edge information to participate in subsequent nonlinear transformations and feature interactions. Element-wise multiplication of the deep semantic features with the first fusion weight matrix and element-wise multiplication of the shallow spatial features with the second fusion weight matrix achieves pixel-level adaptive feature selection, effectively solving the semantic mismatch problem in multi-scale feature fusion. The result of the multiplication is then multiplied by a learnable scalar parameter and added, introducing a controllable edge enhancement mechanism that strengthens target boundary features while avoiding the overfitting risk associated with fixed gains. This fusion strategy significantly improves the jagged edges and blurring issues at object edges in the segmentation results, enhances the recognition accuracy of small targets and complex boundary objects, and strengthens the model's robustness in low-contrast or high-noise environments.

[0078] In one embodiment, in step S40, i.e., the training process of the method, a composite loss function is used, which consists of a cross-entropy loss term, a Dice loss term, and a boundary-aware loss term. The calculation process of the boundary-aware loss term includes: performing morphological gradient operations on the predicted segmentation result and the real label respectively to obtain a predicted boundary map and a real boundary map; and calculating the mean square error between the predicted boundary map and the real boundary map as the boundary-aware loss value.

[0079] Specifically, in this embodiment, the training process of the method employs a composite loss function to comprehensively optimize model performance. This composite loss function consists of a cross-entropy loss term, a Dice loss term, and a boundary-aware loss term. The cross-entropy loss term primarily measures the difference between the predicted probability distribution and the true label distribution, forcing the model to achieve a high confidence level in pixel classification tasks. The Dice loss term focuses on evaluating the consistency of spatial overlap between the predicted and true regions, effectively addressing the training bias problem caused by the imbalance of positive and negative samples, ensuring that the model can accurately segment the main body of the target object. The boundary-aware loss term, as a supplement, specifically optimizes the segmentation accuracy of object edges, compensating for the shortcomings of the first two loss functions in focusing on boundary details.

[0080] The calculation of the boundary-aware loss term first involves the explicit extraction of boundary information. The system performs morphological gradient operations on the predicted segmentation results and the ground truth labels to obtain predicted boundary maps and ground truth boundary maps, respectively. Morphological gradient operations are implemented by calculating the difference between the image dilation and erosion operations. For the predicted segmentation results, this operation highlights the edge contours of regions identified as foreground by the model, generating a predicted boundary map containing the model's predicted boundary locations. For the ground truth labels, this operation extracts manually labeled edges of real objects, generating a ground truth boundary map as the gold standard. Both boundary maps are binary images or probability maps, where highlighted areas represent the contours of objects, and underhighlighted areas represent the background or the interior of objects.

[0081] Subsequently, the system quantifies the deviation between the predicted boundary and the true boundary. The system calculates the mean squared error (MSE) between the predicted and true boundary maps as the boundary-aware loss value. The MSE is calculated by subtracting the value of the corresponding pixel in the true boundary map from the value of each pixel in the predicted boundary map, squaring the difference, and then summing and averaging the squared differences of all pixels. This value directly reflects the degree of spatial deviation between the predicted and true contours. If the predicted boundary highly overlaps with the true boundary, the MSE approaches zero; if the predicted boundary shifts, breaks, or produces false edges, the MSE increases significantly. This boundary-aware loss value is weighted and incorporated into the total composite loss function, guiding the model to focus on correcting parameters in edge regions during backpropagation.

[0082] This invention employs a composite loss function consisting of cross-entropy loss, Dice loss, and boundary-aware loss, achieving comprehensive constraints from pixel classification accuracy and region overlap to edge fit, avoiding optimization dead ends that may occur with a single loss function. Morphological gradient operations are performed on both the predicted segmentation results and the ground truth labels, transforming the abstract segmentation mask into an intuitive boundary representation. This allows the loss function to directly "see" and measure edge errors, enhancing the model's ability to perceive geometric structures. The mean squared error between the predicted and ground truth boundary maps is calculated as the boundary-aware loss value, providing a smooth and differentiable optimization objective that can sensitively penalize minute boundary offsets. This mechanism encourages the model to actively learn sharp edge features during training, significantly reducing common edge blurring, jagged artifacts, and small object loss phenomena in the segmentation results. The resulting model not only performs stably in large-area region segmentation but also achieves extremely high accuracy in finely depicting object contours, making it particularly suitable for medical image analysis or high-precision industrial inspection scenarios with stringent boundary localization requirements.

[0083] In one embodiment, such as Figure 2 As shown, a lightweight real-time semantic segmentation visual recognition system is provided. This lightweight real-time semantic segmentation visual recognition system corresponds one-to-one with the lightweight real-time semantic segmentation visual recognition method in the above embodiments. The lightweight real-time semantic segmentation visual recognition system includes an image acquisition module, a processor module, and an output module. The functional modules are described in detail below: The image acquisition module is used to acquire the image to be processed; The processor module is used to input the image to be processed into the dual-stream heterogeneous coding network, use the dynamic multi-scale sparse attention module to perform sparse attention calculation on the semantic context branch features, use the adaptive gated feature fusion module to perform weighted fusion and edge enhancement processing on the spatial detail branch features and semantic context branch features, and output pixel-level semantic segmentation results. The output module is used to display or transmit the pixel-level semantic segmentation results.

[0084] Furthermore, the processor module includes an embedded graphics processor or a neural network acceleration unit; the embedded graphics processor or neural network acceleration unit is configured to perform depthwise separable convolution operations of the spatial detail branch and sparse matrix multiplication operations of the semantic context branch in parallel; the system also includes a storage module for storing pre-trained weight parameters and intermediate feature map data.

[0085] Specific limitations regarding the lightweight real-time semantic segmentation and visual recognition system can be found in the limitations of the lightweight real-time semantic segmentation and visual recognition method described above, and will not be repeated here. Each module in the aforementioned lightweight real-time semantic segmentation and visual recognition system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0086] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0087] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A lightweight real-time semantic segmentation visual recognition method, characterized in that, include: The image to be processed is acquired and input into a dual-stream heterogeneous coding network, which includes a spatial detail branch that maintains high resolution and a semantic context branch that performs step-downsampling. Dynamic multi-scale sparse attention modules are embedded at different stages of the semantic context branch. The dynamic multi-scale sparse attention modules are used to generate spatial location importance scoring maps and filter key anchor locations. Global attention calculation is performed only on the key anchor locations, and depth convolution operation is performed on non-key anchor locations. The output features of the spatial detail branch and the semantic context branch are input into the adaptive gated feature fusion module. The adaptive gated feature fusion module is used to generate spatial adaptive fusion weights and perform weighted fusion and edge enhancement processing on the two features. The fused features are upsampled by the decoder to output pixel-level semantic segmentation results. 2.The lightweight real-time semantic segmentation visual recognition method of claim 1, wherein, The process of generating a spatial location importance score map by the dynamic multi-scale sparse attention module includes: performing a global average pooling operation on the input feature map to obtain a global feature vector; passing the global feature vector sequentially through a first convolutional layer, a nonlinear activation function, and a second convolutional layer to obtain a scalar score; and mapping the scalar score to the spatial location importance score map through a sigmoid activation function. 3.The lightweight real-time semantic segmentation visual recognition method of claim 2, wherein, The process of the dynamic multi-scale sparse attention module to select key anchor points includes: sorting all pixel values ​​in the spatial location importance score map; selecting a preset number of pixel positions with the largest sorted values ​​as key anchor points; generating a binary mask with the same size as the input feature map; setting the mask value corresponding to the key anchor point position as the first value; and setting the mask value corresponding to the other positions as the second value. 4.The lightweight real-time semantic segmentation visual recognition method of claim 3, wherein, The process of the dynamic multi-scale sparse attention module performing global attention calculation on key anchor points includes: projecting the input feature map through the query weight matrix, key weight matrix, and value weight matrix to obtain the query matrix, key matrix, and value matrix, respectively; for each key anchor point, calculating the dot product of the query vector at that position and the key vectors at all positions, dividing by the scaling factor, and then performing a softmax operation to obtain the attention weights; and weighted summing the attention weights with the value vectors at the corresponding positions to obtain the attention output features for that key anchor point.

5. The lightweight real-time semantic segmentation and visual recognition method according to claim 4, characterized in that, The process by which the dynamic multi-scale sparse attention module performs depthwise convolution operations on non-critical anchor positions includes: identifying all non-critical anchor positions whose values ​​in the binary mask are the second value; directly processing the input features corresponding to the non-critical anchor positions through a depthwise separable convolutional layer to obtain the output features of the non-critical regions; combining the attention output features of the critical anchor positions with the output features of the non-critical regions and obtaining the final output of the module through a channel recovery convolutional layer.

6. The lightweight real-time semantic segmentation and visual recognition method according to claim 1, characterized in that, The process of generating spatial adaptive fusion weights by the adaptive gated feature fusion module includes: concatenating deep semantic features from the semantic context branch with shallow spatial features from the spatial detail branch in the channel dimension; extracting spatial features from the concatenated features through a deep convolutional layer; and mapping the output of the deep convolutional layer through a sigmoid activation function to obtain a first fusion weight matrix and a second fusion weight matrix.

7. The lightweight real-time semantic segmentation and visual recognition method according to claim 6, characterized in that, The adaptive gated feature fusion module performs weighted fusion and edge enhancement processing as follows: using the Sobel operator to perform edge detection on the shallow spatial features to obtain an edge feature map; mapping the edge feature map to the same number of channels through a point convolutional layer to obtain an edge enhancement term; multiplying the deep semantic features element-wise with the first fusion weight matrix, multiplying the shallow spatial features element-wise with the second fusion weight matrix, and adding the result of the multiplication to the edge enhancement term multiplied by a learnable scalar parameter to obtain the fused features.

8. The lightweight real-time semantic segmentation and visual recognition method according to claim 1, characterized in that, The training process of the method employs a composite loss function, which consists of a cross-entropy loss term, a Dice loss term, and a boundary-aware loss term. The calculation process of the boundary-aware loss term includes: performing morphological gradient operations on the predicted segmentation results and the ground truth labels to obtain a predicted boundary map and a ground truth boundary map, respectively; and calculating the mean square error between the predicted boundary map and the ground truth boundary map as the boundary-aware loss value.

9. A lightweight real-time semantic segmentation and visual recognition system, characterized in that, include: The image acquisition module is used to acquire the image to be processed; The processor module is used to input the image to be processed into the dual-stream heterogeneous coding network, use the dynamic multi-scale sparse attention module to perform sparse attention calculation on the semantic context branch features, use the adaptive gated feature fusion module to perform weighted fusion and edge enhancement processing on the spatial detail branch features and semantic context branch features, and output pixel-level semantic segmentation results. The output module is used to display or transmit the pixel-level semantic segmentation results.

10. The lightweight real-time semantic segmentation visual recognition system according to claim 9, characterized in that, The processor module includes an embedded graphics processor or a neural network acceleration unit; the embedded graphics processor or neural network acceleration unit is configured to perform depthwise separable convolution operations of the spatial detail branch and sparse matrix multiplication operations of the semantic context branch in parallel; the system also includes a storage module for storing pre-trained weight parameters and intermediate feature map data.

Images