Semantic segmentation model training method and device

Abstract

The invention relates to a semantic segmentation model training method and device, and belongs to the technical field of digital image.The method comprises the steps that image data on a street are obtained, and an initial semantic segmentation model is set; training the initial semantic segmentation model according to the image data to obtain a target semantic segmentation model and an output result; performing weight calculation according to the initial mask, the pseudo tag and the key edge region set to obtain the total loss of the model; when the total loss of the model is greater than a preset total loss threshold value, expanding the image data according to the pseudo tag, training the target semantic segmentation model according to the expanded image data until the total loss of the model is not greater than the preset total loss threshold value, and determining that model training is completed; according to the method, the initial semantic segmentation model is provided, each image can be processed, and each image does not need to be manually marked and processed; and the output result can be calculated and judged, so that the accuracy of the semantic segmentation model is ensured, and the labor cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of digital image technology, and in particular to a semantic segmentation model training method and device. Background Art

[0002] The application of semantic segmentation technology in automatic parking systems is not limited to the detection of parking spaces and obstacles. It can also be combined with other sensor data (such as lidar, ultrasonic sensors, etc.) to achieve multimodal data fusion and provide the system with richer and more accurate environmental information. For example, in low-light or nighttime environments, semantic segmentation combined with image data from infrared sensors can effectively improve the accuracy of object detection in parking scenes, thereby ensuring that the vehicle can work stably under different lighting conditions. With the continuous advancement of autonomous driving technology, the application of semantic segmentation in autonomous driving and automatic parking systems is becoming more and more extensive and important. It not only helps autonomous vehicles achieve accurate target recognition in complex environments, but also ensures the safety and efficiency of automatic parking operations.

[0003] In the existing technology, when performing semantic segmentation tasks, annotators are required to manually label each image and collect accurately labeled data for training deep neural networks, which greatly increases the labor cost. In addition, for certain specific semantic categories, labeled samples may be difficult to obtain, resulting in limited model performance.

[0004] Therefore, it is urgent to propose a semantic segmentation model training method and device to solve the technical problems that the semantic segmentation method in the prior art requires manual labeling of each image, and the labeled samples may be difficult to obtain, resulting in increased labor costs and labor costs. Summary of the invention

[0005] In view of this, it is necessary to provide a semantic segmentation model training method and device to solve the technical problems that the semantic segmentation method in the prior art requires manual labeling of each image, and the labeled samples may be difficult to obtain, resulting in increased labor costs and labor costs.

[0006] In order to solve the above problems, the present invention provides a semantic segmentation model training method, comprising: Obtain image data on the street and set up the initial semantic segmentation model; Training the initial semantic segmentation model according to the image data to obtain a target semantic segmentation model and an output result; Performing weight calculation according to the initial mask, the pseudo label and the key edge region set to obtain a total model loss; When the total loss of the model is greater than a preset total loss threshold, the image data is expanded according to the pseudo-label, and the target semantic segmentation model is trained according to the expanded image data until the total loss of the model is no greater than the preset total loss threshold, and the model training is determined to be completed.

[0007] In a possible implementation, the training of the initial semantic segmentation model according to the image data to obtain a target semantic segmentation model and an output result includes: Annotating the image data to obtain a data set; Dividing the data set to obtain a training set; The training set is input into the initial semantic segmentation model for training to obtain a target semantic segmentation model and output results.

[0008] In a possible implementation, the initial semantic segmentation model includes a feature extraction network, a perception layer, an action layer, and a feedback layer. The training set is input into the initial semantic segmentation model for training to obtain a target semantic segmentation model and an output result, including: Inputting the image data of the training set into the feature extraction network for layer-by-layer sampling to obtain a feature map; Inputting the feature map into the perception layer to extract key areas and obtain key area features; Inputting the feature map and the key area features into the action layer for mask extraction and dynamic optimization to obtain an initial mask and a dynamic weight map; The initial mask and the dynamic weight map are input into the feedback layer to screen high confidence areas to obtain an output result.

[0009] In a possible implementation, the output result includes an initial mask, a pseudo label, and a set of key edge regions; and performing weight calculation according to the initial mask, the pseudo label, and the set of key edge regions to obtain a total model loss includes: Performing standard forward propagation and backward propagation calculations on the initial mask and the pseudo-label to obtain a first weight; Obtaining a second weight and a third weight according to the key edge region set and the pseudo-label; The first weight, the second weight, and the third weight are calculated to obtain a total model loss.

[0010] In a possible implementation manner, obtaining the second weight and the third weight according to the key edge region set and the pseudo label includes: Performing mask extraction on the key edge region set to obtain an edge region mask; Performing edge region loss calculation on the edge region mask and the pseudo label to obtain a second weight; The edge region mask, the pseudo label and the preset weight are balanced and optimized to obtain a third weight.

[0011] In a possible implementation, the feature extraction network includes a mixed frequency domain convolution layer and a multi-scale attention residual module; the image data of the training set is input into the feature extraction network for layer-by-layer sampling to obtain a feature map, including: Inputting the image data into the mixed frequency domain convolution layer for weight calculation to obtain an adaptive weight for each feature channel; Determine the convolution mode corresponding to each feature channel according to the adaptive weight; the convolution mode whose adaptive weight is greater than the preset weight threshold is the traditional convolution; the convolution mode whose adaptive weight is not greater than the preset weight threshold is the frequency domain convolution; Performing feature extraction on the image data according to the convolution mode to obtain a feature image of each feature channel; The feature image is subjected to multi-scale feature fusion and adjustment according to the multi-scale attention residual module to obtain a feature map.

[0012] In a possible implementation, the step of inputting the feature map into the perception layer to extract key areas to obtain key area features includes: Inputting the output features of each feature channel of the feature map into the perception layer for global statistics to obtain the global response strength of each feature channel; Determine the feature channel whose global response intensity is greater than a preset intensity threshold as a key feature channel, and determine the feature of the key feature channel as a local feature; The global context information and the local features are calculated according to the global average pooling layer to obtain the global features; The local feature is merged with the global feature to obtain an enhanced feature; The enhanced features are screened based on a dynamic gating mechanism to obtain key area features.

[0013] In a possible implementation, the first weight is calculated as follows:

[0014]

[0015]

[0016]

[0017] In the formula, The initial mask Middle i The mask data of For the i Pseudo labels, The initial mask The pixel propagation method is used to obtain i Pixels propagate data, For the i pixels, is the first weight.

[0018] In a possible implementation, the second weight is calculated as follows:

[0019] In the formula, is the total number of edge area pixels in the edge area mask; is the cross entropy loss function; is the first Edge values; For the Pseudo-label.

[0020] On the other hand, the present invention also provides a semantic segmentation model training device, comprising: The data acquisition module is used to acquire image data on the street and set up an initial semantic segmentation model; A model training module, used to train the initial semantic segmentation model according to the image data to obtain a target semantic segmentation model and output results; A weight calculation module, used for performing weight calculation according to the initial mask, the pseudo label and the key edge region set to obtain a total model loss; The result judgment module is used to expand the image data according to the pseudo-label when the total loss of the model is greater than the preset total loss threshold, and train the target semantic segmentation model according to the expanded image data until the total loss of the model is no greater than the preset total loss threshold, thereby determining that the model training is completed.

[0021] The beneficial effects of the present invention are as follows: acquiring image data on the street and setting an initial semantic segmentation model; training the initial semantic segmentation model according to the image data to obtain a target semantic segmentation model and an output result; the output result includes an initial mask, a pseudo-label and a set of key edge regions; performing weight calculation according to the initial mask, the pseudo-label and the set of key edge regions to obtain a total model loss; when the total model loss is greater than a preset total loss threshold, expanding the image data according to the pseudo-label, and training the target semantic segmentation model according to the expanded image data until the total model loss is no greater than the preset total loss threshold, and determining that the model training is completed; the present invention provides an initial semantic segmentation model that can process each image without manually marking and processing each image; further, the output result can also be calculated and judged, thereby ensuring the accuracy of the semantic segmentation model and reducing labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of an embodiment of the semantic segmentation model training method provided by the present invention; Figure 2 For the present invention Figure 1 A schematic flow chart of an embodiment of step S102; Figure 3 For the present invention Figure 2 A schematic flow chart of an embodiment of step S203; Figure 4 For the present invention Figure 3 A schematic flow chart of an embodiment of step S301; Figure 5 For the present invention Figure 3 A schematic flow chart of an embodiment of step S302; Figure 6 For the present invention Figure 1 A schematic flow chart of an embodiment of step S103; Figure 7 A schematic diagram of the structure of an embodiment of the model visualization result provided by the present invention; Figure 8 A schematic diagram of the structure of an embodiment of a semantic segmentation model training device provided by the present invention; Fig. 9 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0024] like Figure 1As shown, a specific embodiment of the present invention discloses a semantic segmentation model training method, comprising: S101, obtaining image data on the street and setting an initial semantic segmentation model; S102, training the initial semantic segmentation model according to the image data to obtain a target semantic segmentation model and an output result; the output result includes an initial mask, a pseudo label, and a set of key edge regions; S103, performing weight calculation according to the initial mask, pseudo label and key edge region set to obtain the total loss of the model; S104. When the total loss of the model is greater than the preset total loss threshold, the image data is expanded according to the pseudo-label, and the target semantic segmentation model is trained according to the expanded image data until the total loss of the model is no greater than the preset total loss threshold, and the model training is determined to be completed.

[0025] It should be understood that the image data acquired in step S101 may be images of various streets or other places acquired by an image acquisition device, or may be images historically stored in a storage medium.

[0026] In a specific embodiment of the present invention, the image acquisition device can be used to take pictures in various corners of the street, which can cover various environmental conditions. By taking pictures in different time periods and environments (such as day, night, sunny days, rainy days, etc.), the diversity and representativeness of the data are ensured. These captured data will be used for subsequent semantic segmentation analysis. An initial semantic segmentation model can also be set. The initial semantic segmentation model can be a PAF-MSM model, that is, a model constructed by a perception-action-feedback loop (PAF) and a traditional neural network (MSM). In order to improve the accuracy of the semantic segmentation of the initial semantic segmentation model, the initial semantic segmentation model can be trained with image data to obtain the output results of the initial semantic segmentation model and the trained target semantic segmentation model, wherein the output results include an initial mask, a pseudo label, and a set of key edge regions. In order to train the models separately, the initial mask can be trained. Processing is performed to filter out high confidence areas for generating pseudo labels , for each pixel i , if the maximum confidence of its prediction Max( ) is higher than the set threshold τ , then its predicted category is used as a pseudo label , otherwise the pixel is ignored. Gradient calculation can also be performed on the training set to obtain the key edge region set Edge.

[0027] In order to determine whether the target semantic segmentation model meets the requirements, weight calculation can be performed based on the initial mask, pseudo-label and key edge area set to obtain the weight of each calculation, so as to calculate the total loss of the model. Then, the judgment result of the total loss of the model and the preset total loss threshold can be used to determine whether the target semantic segmentation model meets the training requirements. Specifically, if the total loss of the model is greater than the preset total loss threshold, the image data can be expanded according to the pseudo-label, and the target semantic segmentation model can be trained according to the expanded image data until the total loss of the model is no greater than the preset total loss threshold. It is determined that the model training is completed, and then the trained semantic segmentation model can be used to perform semantic segmentation on the image data.

[0028] Compared with the prior art, the present embodiment provides the method of acquiring image data on the street and setting an initial semantic segmentation model; training the initial semantic segmentation model according to the image data to obtain a target semantic segmentation model and an output result; the output result includes an initial mask, a pseudo-label and a set of key edge areas; weight calculation is performed according to the initial mask, the pseudo-label and the set of key edge areas to obtain a total model loss; when the total model loss is greater than a preset total loss threshold, the image data is expanded according to the pseudo-label, and the target semantic segmentation model is trained according to the expanded image data until the total model loss is no greater than the preset total loss threshold, and the model training is determined to be completed; the present invention provides an initial semantic segmentation model, which can process each image without manually marking and processing each image; further, the output result can also be calculated and judged, thereby ensuring the accuracy of the semantic segmentation model and reducing labor costs.

[0029] In some embodiments of the present invention, Figure 2 As shown, step S102 includes: S201, annotating image data to obtain a data set; S202, dividing the data set to obtain a training set; S203: Input the training set into the initial semantic segmentation model for training to obtain a target semantic segmentation model and output results.

[0030] In a specific embodiment of the present invention, after the image data is collected, the image data can be annotated, for example, pedestrians, cars, traffic lights, zebra crossings and other road scene images can be annotated, so that the annotated image data can be obtained, and all the image data can be combined into a data set, and then the data set can be divided according to a preset ratio. For example, the data set has 20,000 road scene pictures, and the preset ratio can be 8:1:1, then the corresponding number of training sets, verification sets and test sets can be obtained, and then the image data in the training set can be input into the initial semantic segmentation model, and the initial semantic segmentation model can be trained, so as to obtain the target semantic segmentation model and output results.

[0031] In some embodiments of the present invention, the initial semantic segmentation model includes a feature extraction network, a perception layer, an action layer, and a feedback layer, such as Figure 3 As shown, step S203 includes: S301, inputting the image data of the training set into the feature extraction network for layer-by-layer sampling to obtain a feature map; S302, inputting the feature map into the perception layer to extract the key area and obtain the key area features; S303, inputting the feature map and key area features into the action layer for mask extraction and dynamic optimization to obtain an initial mask and a dynamic weight map; S304: Input the initial mask and the dynamic weight map into the feedback layer to screen the high confidence area and obtain the output result.

[0032] In a specific embodiment of the present invention, the initial semantic segmentation model may include a feature extraction network, a perception layer, an action layer and a feedback layer. The image data of the training set may be input into the feature extraction network for layer-by-layer sampling to obtain a feature map. Specifically, in some embodiments of the present invention, the feature extraction network includes a mixed frequency domain convolution layer and a multi-scale attention residual module; Figure 4 As shown, step S301 includes: S401, inputting the image data into the mixed frequency domain convolution layer for weight calculation to obtain the adaptive weight of each feature channel; S402, determining the convolution mode corresponding to each feature channel according to the adaptive weight; the convolution mode whose adaptive weight is greater than the preset weight threshold is the traditional convolution; the convolution mode whose adaptive weight is not greater than the preset weight threshold is the frequency domain convolution; S403, extracting features from the image data according to the convolution mode to obtain a feature image of each feature channel; S404, performing multi-scale feature fusion and adjustment on the feature image according to the multi-scale attention residual module to obtain a feature map.

[0033] In a specific embodiment of the present invention, the feature extraction network may include a hybrid frequency domain convolution layer and a multi-scale attention residual module. The feature extraction network adopts a hybrid frequency domain convolution (HFDC) neural network structure, and integrates a customized multi-scale attention residual module (MSARM) with different convolution scale features to enhance the feature extraction capability, wherein the hybrid frequency domain convolution (HFDC) optimizes the model performance according to the feature complexity and hierarchical requirements by adaptively selecting traditional convolution and frequency domain convolution. At the shallow level, traditional convolution is suitable for processing local features and is computationally efficient; at the deep level, frequency domain convolution effectively compresses the amount of calculation through frequency domain transformation, and captures long-distance dependencies, overcoming the limitations of traditional convolution. The HFDC module can adaptively switch convolution modes at different layers, improve feature expression, and optimize computational efficiency. Specifically, before each convolution, the HFDC module calculates adaptive weights based on the information of the input image data. , calculated as shown in formula (1): = (Net( ))(1) In the formula, Net( ) is based on the input image data The network calculation results are The sigmoid function limits the output to the range [0, 1].

[0034] This weight is learned through the network and determines whether to use traditional convolution or frequency domain convolution. If the complexity of the feature map is low (such as shallow features), select traditional convolution; if the complexity of the feature map is high (such as deep features), select frequency domain convolution. Through this mechanism, the HFDC module can automatically select convolution modes at different levels to ensure that the feature channels of each layer can extract features from the image data in the best way, so that the feature image of each feature channel can be obtained. The multi-scale attention residual module can then perform multi-scale feature fusion and adjustment on the feature image to obtain the feature map. Among them, the MSARM module (Multi-Scale Attention Residual Module) aims to enhance the feature extraction capability through multi-scale feature fusion and dynamic weighting mechanism, which is suitable for processing feature expressions of complex scenes. Input features and multi-scale features a1 , a2 , a3 As the input of the module, semantic information under different receptive fields is provided respectively. In the first step, multi-scale features are integrated through the feature fusion module (Feature Fusion) to obtain preliminary fusion features The fusion process can be achieved through splicing or weighted summation to ensure the synergy of information at different scales in subsequent processing. Then it enters the pooling layer for spatial compression to enhance the perception of global features. After pooling, the features are reduced in dimension through a 1×1 convolution layer to reduce computational cost and introduce initial nonlinearity. Then, the features are passed through another layer of 1×1 convolution to adapt to the processing requirements of the next stage. An important link is the weighted feature fusion mechanism, which combines the processed features with the original input features. Through the attention mechanism or dynamic weight adjustment, the module can identify the importance of features and thus optimize the fusion effect. The fused features are further normalized and nonlinearly enhanced through batch normalization and activation function (ReLU). Finally, the module introduces residual connection to directly convert the input features into The output is introduced to alleviate the gradient vanishing problem and retain the information of the original features. The final output features after fusion combine multi-scale, global context and original input features to obtain feature maps, which not only enrich the feature expression but also maintain the fluidity and consistency of the data. Through this process, the MSARM module has strong adaptability and expressiveness in complex feature extraction tasks.

[0035] Furthermore, the MSM network consists of a feature extraction network MS and a weighted fusion operation, in which all convolutions are replaced by mixed frequency convolution modules. The MS extraction network is a multi-scale, multi-dimensional network. The cross-feature fusion part performs multiple expansion convolution operations on the deep information, which greatly optimizes and balances the receptive field. Finally, it performs a concat operation with the shallow features to output the result. For example, the size of the input image data is 1024×1024×C, where C represents the number of channels, and C=3 for RGB images. First, the input is sent to three parallel branches, which use 1x1, 3x3, and 5x5 mixed frequency convolutions for feature extraction respectively. The 1×1 convolution retains the spatial resolution of the image while realizing the linear combination of channel features. The output is ; 3×3 and 5×5 convolutions provide complementarity in extracting small-scale and large-scale features, and the outputs are and Then, these outputs are combined with the input of the previous layer It is input into the multi-convolutional scale feature attention module (MSARM), in which the multi-scale features are weightedly fused through the attention mechanism to obtain the output , the spatial resolution is still 1024×1024. Then, the backbone structure of the network begins to process the features at multiple levels. The downsampled image is then downsampled (convolution and pooling operations with a stride of 2×2) to a resolution of 512×512. The downsampled features are extracted by another set of parallel 1×1, 3×3, and 5×5 convolution kernels and then sent to the MSARM module for feature fusion to obtain the mid-scale features. Next, these features are downsampled again (resolution is halved to 256×256), and the above operations are repeated to extract features, resulting in a deeper multi-scale fusion result. , , .

[0036] After deep-level feature extraction is completed, the embodiment of the present invention adopts a weighted fusion method for feature maps of different levels, and the size of the weight reflects the contribution of the features of this layer. Weighted fusion adopts the following method: (2) in, For the i The feature map of the layer, N is the number of pixels, is the adaptive weight of the feature map of this layer, is the final fused feature map. Through this weighted summation, the model can dynamically adjust the importance of different semantic levels.

[0037] In some embodiments of the present invention, Figure 5 As shown, step S302 includes: S501, inputting the output features of each feature channel of the feature map into the perception layer for global statistics to obtain the global response strength of each feature channel; S502, determining a feature channel whose global response intensity is greater than a preset intensity threshold as a key feature channel, and determining the features of the key feature channel as local features; S503, calculating the global context information and the local features according to the global average pooling layer to obtain the global features; S504, fusing the local features with the global features to obtain enhanced features; S505: Screen the enhanced features based on the dynamic gating mechanism to obtain key area features.

[0038] In a specific embodiment of the present invention, the perception layer is a key part of the entire MSM network, which is intended to extract key areas from the high-dimensional features output by the feature extraction network to support the operation of the subsequent action layer and feedback module. The specific steps may be: the output feature of each feature channel of the output feature map of the feature extraction network MS is F∈ Perform global statistical analysis to calculate the importance of each feature channel. Use Global Average Pooling (GAP) to calculate the global response strength of each feature channel. , calculated as shown in formula (2): (2) In the formula, For Channel c The importance score reflects the contribution of the feature channel to the target task. C is the number of feature channels, H and W are the height and width of the feature map of the feature channel respectively.

[0039] The global response intensity of each feature channel can then be compared with a preset intensity threshold T For comparison, you can Above a preset intensity threshold T , then the feature channel is considered to contain the information of the key area, that is, it satisfies c ∈{ c | >T The feature channels with global response intensity greater than the preset intensity threshold can be retained, and the feature channels with global response intensity greater than the preset intensity threshold can be determined as key feature channels to complete the selection of key feature channels. Then the perception layer can further analyze the spatial distribution of the features by calculating the spatial distribution of each pixel. N The attention weights are used to highlight the key areas and suppress the background. The weight calculation of spatial attention is shown in formula (3): (3) In the formula, and b is a learnable parameter, is the activation function (Sigmoid), and represents the characteristics of the key feature channel. , then the corresponding pixel is considered to belong to the key area, that is, the local feature, and the pixel selection condition is .

[0040] To further enhance the recognition of key areas, the perception layer combines global context information. The global features are calculated through the global average pooling layer GAP operation. , calculated as shown in formula (4): (4) In the formula, It is the feature map of local features.

[0041] Then the local features can be fused with the global features to obtain enhanced features. , calculated as shown in formula (5): (5) This fusion process ensures that local features are more accurate under the guidance of global semantic consistency, further optimizing the expressiveness of features. Finally, the perception layer screens the enhanced features through a dynamic gating mechanism, adjusts the information flow in feature channels and space, and dynamically controls the gating weights. The calculation of is shown in formula (6): (6) The characteristic representation after the dynamic gating mechanism is: , The dynamic gating mechanism dynamically adjusts the degree of feature selection according to task requirements. , then filter out the corresponding features. Through this multi-level feature analysis and screening, the perception layer effectively locates the key areas and obtains the filtered key area features. , which lays the foundation for generating high-quality initial masks for subsequent action layers.

[0042] The design of the action layer is closely integrated with the output of the perception layer. It supports the improvement of the overall model performance by fusing key area features with global features, generating initial masks, and dynamically optimizing segmentation decisions. First, the action layer receives the key area features generated by the perception layer. and the features of the original feature extraction network The preliminary fusion features are generated by weighted fusion calculation, as shown in formula (7): (7) in, α ∈[0,1] is a weight parameter used to control the balance between key features and global features. This feature fusion method can not only highlight the key area information screened by the perception layer, but also ensure that the global background information is not ignored. , the action layer generates the initial mask through the weighted fusion operation of the MSM network , the decoding branch uses convolution and upsampling modules to Convert to a mask prediction that is consistent with the input image size, as shown in formula (8): (8) In the formula, and are the convolution weights and bias terms respectively.

[0043] The action layer not only generates the initial mask , and dynamically adjust the segmentation decision. Using the key area mask output by the perception layer , generate the weight graph through the dynamic weight balancing formula w ( x , y ), as shown in formula (9): (9) In the formula, γ ∈[0,1] is a dynamic adjustment parameter. If γ Larger, the model focuses more on key areas ,if γ Smaller, the model focuses more on the initial global prediction , calculated as ,in t is the current training round, is the step size of each round of adjustment. Finally, the output of the action layer is the initial mask and dynamic weight graph w ( x , y ). The initial mask is used to provide guidance for subsequent optimization modules, and the dynamic weight map ensures the balance between the critical area and other areas.

[0044] In some embodiments of the present invention, Figure 6 As shown, step S103 includes: S601, performing standard forward propagation and backward propagation calculations on the initial mask and pseudo-label to obtain a first weight; S602, obtaining a second weight and a third weight according to the key edge region set and the pseudo label; S603: Calculate the first weight, the second weight, and the third weight to obtain the total loss of the model.

[0045] In a specific embodiment of the present invention, during the first calculation process, standard forward propagation and back propagation can be performed on the input initial mask and pseudo label, as shown in formulas (10)-(13): (10) (11) (12) (13) In the formula, The initial mask Middle i The mask data of For thei Pseudo labels, The initial mask The pixel propagation method is used to obtain i Pixels propagate data, For the i pixels, is the first weight.

[0046] In some embodiments of the present invention, step S602 includes: Perform mask extraction on the key edge region set to obtain an edge region mask; Calculate the edge region loss for the edge region mask and the pseudo label to obtain a second weight; The edge region mask, pseudo label and preset weight are balanced and optimized to obtain the third weight.

[0047] In a specific embodiment of the present invention, in the second calculation, the edge region loss is optimized, the key edge region set Edge is extracted, and the edge region mask is obtained by the gradient method or the morphological method. , and then the second weight is calculated based on the pixels in the edge area using formula (14): (14) In the formula, is the total number of edge area pixels in the edge area mask; is the cross entropy loss function; is the first Edge values; For the Pseudo-labels, is the second weight.

[0048] In the third weight calculation, we return to the balanced optimization of the global loss. By combining the loss weights of ordinary pixels and key areas, we calculate the third weight, as shown in formula (15): (15) In the formula, is the weight, calculated as: ,if i / 1, is the weight gain coefficient of the edge area (such as = 0.2), the weight of the loss function The influence of edge pixels and ordinary pixels is balanced to ensure global optimization effect.

[0049] Then the first weight, the second weight, and the third weight can be calculated to obtain the total loss of the model, as shown in formula (16): L= (16) in, As shown in formulas (17)-(19) respectively: (17) (18) (19) In the formula, L is the total loss of the model, T is the total number of training calculations, t For the current round.

[0050] Among them, the above three weight calculations can be performed simultaneously. Through the three weight calculations, the optimization of the edge area and the global loss is refined layer by layer, and the learning rate and weight are adjusted in a formula to give more attention to the key areas while maintaining a balanced improvement in the overall model performance. The combination of the perception-action-feedback loop (PAF) and the traditional neural network (MSM) has jointly promoted the improvement of semantic segmentation performance through their interaction. The PAF loop enables the network to achieve more intelligent decision-making and optimization based on the traditional neural network, overcoming the traditional method that relies on manually designed data enhancement and training steps, and provides a new solution for automated image segmentation tasks.

[0051] Furthermore, it can be determined whether the total loss of the model is greater than the preset total loss threshold. If not, it means that the semantic segmentation model has achieved the ideal effect, and the target semantic segmentation model can be determined to be a trained model. If so, the pseudo-labels can be expanded to the training set to obtain a new training set, and the target semantic segmentation model can be trained according to the new training set until the total loss of the model is not greater than the preset total loss threshold, and the model training is determined to be completed. Specifically, it can be through the initial mask Dynamically adjust the data samples and training set weights to improve the model's ability to segment complex scenes. Afterwards, the pseudo labels can be provided The feature maps are stored in the training set to expand the training set, which significantly improves data utilization. The loss weights of the pixels in these areas in the key edge area set Edgez can also be calculated. Afterwards, ensure that these areas account for a larger proportion of the total loss. The embodiment of the present invention also feeds these difficult samples back to the training set to obtain a new training set. In the next round of training, the frequency of these "images with blurred edges" is increased and the training of simple images is reduced. The loop optimization gradually improves the model's processing capabilities for edge areas and complex backgrounds through multiple calculation optimizations of key samples, and balances the losses at the global level.

[0052] Furthermore, after the target semantic segmentation model is trained, it can be evaluated. For example, the target semantic segmentation model PAF-MSM (ours) is compared with the classic semantic segmentation networks UNet, Deeplabv3+ and Segnet to prove the effectiveness of the proposed network. The model visualization results are shown in Figure 2. Figure 7 As shown, according to Figure 7 It can be seen that the IoU, Recall and Precision of the PAF-MSM model are higher than those of other segmentation methods. The PAF-MSM model has achieved very good segmentation results. The PAF-MSM method is better than UNet, Deeplabv3+ and Segnet in the segmentation accuracy of industrial road scenes.

[0053] In order to better implement the semantic segmentation model training method in the embodiment of the present invention, based on the semantic segmentation model training method, the embodiment of the present invention also provides a semantic segmentation model training device, such as Figure 8 As shown, the semantic segmentation model training device 800 includes: The data acquisition module 801 is used to acquire image data on the street and set an initial semantic segmentation model; The model training module 802 is used to train the initial semantic segmentation model according to the image data to obtain the target semantic segmentation model and output results; the output results include the initial mask, pseudo label and key edge region set; A weight calculation module 803 is used to perform weight calculation based on the initial mask, pseudo label and key edge region set to obtain the total model loss; The result judgment module 804 is used to expand the image data according to the pseudo-label when the total loss of the model is greater than the preset total loss threshold, and train the target semantic segmentation model according to the expanded image data until the total loss of the model is no greater than the preset total loss threshold, and determine that the model training is completed.

[0054] The semantic segmentation model training device 800 provided in the above embodiment can implement the technical solution described in the above semantic segmentation model training method embodiment. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above semantic segmentation model training method embodiment, which will not be repeated here.

[0055] like Fig. 9 As shown, the present invention also provides an electronic device 900. The electronic device 900 includes a processor 901, a memory 902 and a display 903. Fig. 9 Only some components of the electronic device 900 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0056] In some embodiments, the memory 902 may be an internal storage unit of the electronic device 900, such as a hard disk or memory of the electronic device 900. In other embodiments, the memory 902 may also be an external storage device of the electronic device 900, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the electronic device 900.

[0057] Furthermore, the memory 902 may include both an internal storage unit of the electronic device 900 and an external storage device. The memory 902 is used to store application software installed in the electronic device 900 and various data.

[0058] In some embodiments, the processor 901 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 902, such as the semantic segmentation model training method of the present invention.

[0059] In some embodiments, the display 903 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 903 is used to display information of the electronic device 900 and to display a visual user interface. The components 901-903 of the electronic device 900 communicate with each other through a system bus.

[0060] In some embodiments of the present invention, when the processor 901 executes the semantic segmentation model training program in the memory 902, the following steps may be implemented: Obtain image data on the street and set up the initial semantic segmentation model; The initial semantic segmentation model is trained according to the image data to obtain the target semantic segmentation model and output results; the output results include the initial mask, pseudo label and key edge region set; The total loss of the model is obtained by weighting the initial mask, pseudo-label and key edge region set. When the total loss of the model is greater than the preset total loss threshold, the image data is expanded according to the pseudo-label, and the target semantic segmentation model is trained based on the expanded image data until the total loss of the model is no greater than the preset total loss threshold, and the model training is determined to be completed.

[0061] It should be understood that: when the processor 901 executes the semantic segmentation model training program in the memory 902, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0062] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 900 mentioned, and the electronic device 900 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 900 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0063] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the semantic segmentation model training method steps or functions provided in the above-mentioned method embodiments can be implemented.

[0064] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0065] The semantic segmentation model training method and device provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A semantic segmentation model training method, characterized in that: include: Obtain image data on the street and set up the initial semantic segmentation model; Training the initial semantic segmentation model according to the image data to obtain a target semantic segmentation model and an output result; the output result includes an initial mask, a pseudo label, and a set of key edge regions; Performing weight calculation according to the initial mask, the pseudo label and the key edge region set to obtain a total model loss; When the total loss of the model is greater than a preset total loss threshold, the image data is expanded according to the pseudo-label, and the target semantic segmentation model is trained according to the expanded image data until the total loss of the model is no greater than the preset total loss threshold, and the model training is determined to be completed.

2. The semantic segmentation model training method according to claim 1, characterized in that: The initial semantic segmentation model is trained according to the image data to obtain a target semantic segmentation model and an output result, including: Annotating the image data to obtain a data set; Dividing the data set to obtain a training set; The training set is input into the initial semantic segmentation model for training to obtain a target semantic segmentation model and output results.

3. The semantic segmentation model training method according to claim 2, characterized in that: The initial semantic segmentation model includes a feature extraction network, a perception layer, an action layer and a feedback layer. The training set is input into the initial semantic segmentation model for training to obtain a target semantic segmentation model and an output result, including: Inputting the image data of the training set into the feature extraction network for layer-by-layer sampling to obtain a feature map; Inputting the feature map into the perception layer to extract key areas and obtain key area features; Inputting the feature map and the key area features into the action layer for mask extraction and dynamic optimization to obtain an initial mask and a dynamic weight map; The initial mask and the dynamic weight map are input into the feedback layer to screen high confidence areas to obtain an output result.

4. The semantic segmentation model training method according to claim 1, characterized in that: The weight calculation is performed according to the initial mask, the pseudo label and the key edge region set to obtain the total model loss, including: Performing standard forward propagation and backward propagation calculations on the initial mask and the pseudo-label to obtain a first weight; Obtaining a second weight and a third weight according to the key edge region set and the pseudo-label; The first weight, the second weight, and the third weight are calculated to obtain a total model loss.

5. The semantic segmentation model training method according to claim 4, characterized in that: The obtaining of the second weight and the third weight according to the key edge region set and the pseudo label comprises: Performing mask extraction on the key edge region set to obtain an edge region mask; Performing edge region loss calculation on the edge region mask and the pseudo label to obtain a second weight; The edge region mask, the pseudo label and the preset weight are balanced and optimized to obtain a third weight.

6. The semantic segmentation model training method according to claim 3, characterized in that: The feature extraction network includes a mixed frequency domain convolution layer and a multi-scale attention residual module; The step of inputting the image data of the training set into the feature extraction network for layer-by-layer sampling to obtain a feature map includes: Inputting the image data into the mixed frequency domain convolution layer for weight calculation to obtain an adaptive weight for each feature channel; Determine the convolution mode corresponding to each feature channel according to the adaptive weight; the convolution mode whose adaptive weight is greater than the preset weight threshold is the traditional convolution; the convolution mode whose adaptive weight is not greater than the preset weight threshold is the frequency domain convolution; Performing feature extraction on the image data according to the convolution mode to obtain a feature image of each feature channel; The feature image is subjected to multi-scale feature fusion and adjustment according to the multi-scale attention residual module to obtain a feature map.

7. The semantic segmentation model training method according to claim 6, characterized in that: The step of inputting the feature map into the perception layer to extract key areas and obtain key area features includes: Inputting the output features of each feature channel of the feature map into the perception layer for global statistics to obtain the global response strength of each feature channel; Determine the feature channel whose global response intensity is greater than a preset intensity threshold as a key feature channel, and determine the feature of the key feature channel as a local feature; The global context information and the local features are calculated according to the global average pooling layer to obtain the global features; The local feature is merged with the global feature to obtain an enhanced feature; The enhanced features are screened based on a dynamic gating mechanism to obtain key area features.

8. The semantic segmentation model training method according to claim 4, characterized in that: The first weight is calculated as follows: In the formula, The initial mask Middle i The mask data of For the i Pseudo labels, The initial mask The pixel propagation method is used to obtain i Pixels propagate data, For the i pixels, is the first weight.

9. The semantic segmentation model training method according to claim 5, characterized in that: The second weight is calculated as follows: In the formula, is the total number of edge area pixels in the edge area mask; is the cross entropy loss function; is the first Edge values; For the Pseudo-label.

10. A semantic segmentation model training device, characterized in that: include: The data acquisition module is used to acquire image data on the street and set up an initial semantic segmentation model; A model training module, used to train the initial semantic segmentation model according to the image data to obtain a target semantic segmentation model and an output result; the output result includes an initial mask, a pseudo label and a set of key edge regions; A weight calculation module, used for performing weight calculation according to the initial mask, the pseudo label and the key edge region set to obtain a total model loss; The result judgment module is used to expand the image data according to the pseudo-label when the total loss of the model is greater than the preset total loss threshold, and train the target semantic segmentation model according to the expanded image data until the total loss of the model is no greater than the preset total loss threshold, thereby determining that the model training is completed.

Images