Semantic segmentation model deployment method and device, storage medium and equipment

Abstract

The invention discloses a semantic segmentation model deployment method and device, a storage medium and equipment, and belongs to the technical field of automatic driving. Creating a semantic segmentation model suitable for DLA; obtaining a training set; for each training sample, processing a sample image by using a semantic segmentation model to obtain a prediction result, and performing loss calculation on the prediction result, a real category label and a real boundary label by using a detail loss function, three semantic loss functions, a boundary loss function and two segmentation loss functions to obtain a real category label and a real boundary label; training model parameters of a semantic segmentation model according to a calculation result; performing INT8 quantitative training on the trained semantic segmentation model to convert the semantic segmentation model from a PyTorch format to an ONNX format; and deploying the semantic segmentation model in the ONNX format to the DLA. According to the method, the semantic segmentation model is migrated from the GPU to the DLA, and the reasoning speed of the semantic segmentation model is optimized on the premise that the precision is not obviously sacrificed.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method, apparatus, storage medium and device for deploying a semantic segmentation model. Background Art

[0002] The semantic segmentation model is an image segmentation technology in the field of computer vision. Its core task is to assign each pixel in the image to a predefined semantic category, which can be vehicles, pedestrians, buildings, etc.

[0003] The normal operation of the perception algorithms of autonomous vehicles is inseparable from the support of embedded artificial intelligence chips, such as Jetson Orin. Typically, these chips are based on the Graphics Processing Unit (GPU) in the Nvidia Ampere architecture. In order to ensure that the semantic segmentation model can respond correctly and quickly, the perception algorithm containing the semantic segmentation model is usually deployed on the GPU.

[0004] However, as the number of models deployed on GPUs increases, multiple models need to compete for computing resources, affecting the performance of semantic segmentation models. Summary of the invention

[0005] The present application provides a method, apparatus, storage medium and device for deploying a semantic segmentation model, which is used to solve the problem that when a semantic segmentation model is deployed on a GPU, multiple models need to compete for computing resources, which affects the performance of the semantic segmentation model. The technical solution is as follows: According to a first aspect of the present application, a method for deploying a semantic segmentation model is provided, the method comprising: Creating a semantic segmentation model suitable for a deep learning accelerator DLA, the semantic segmentation model comprising a feature extraction module, a detail extraction branch, a semantic extraction branch, a boundary extraction branch, a fusion module, and an upsampling segmentation head, wherein the detail prediction head in the detail extraction branch corresponds to a detail loss function, the two auxiliary heads and the optimized feature pyramid in the semantic extraction branch correspond to a semantic loss function respectively, the boundary prediction head in the boundary extraction branch corresponds to a boundary loss function, and the upsampling segmentation head corresponds to two segmentation loss functions with different sampling rates; Obtaining a training set, wherein the training samples in the training set include sample images, true category labels, and true boundary labels, wherein the true boundary labels are obtained by performing boundary extraction on the true category labels; For each training sample, the sample image is processed using the semantic segmentation model to obtain a prediction result, and the prediction result, the true category label and the true boundary label are calculated using the detail loss function, three semantic loss functions, the boundary loss function and two segmentation loss functions, and the model parameters of the semantic segmentation model are trained according to the calculation results; Perform INT8 quantization training on the trained semantic segmentation model to convert the semantic segmentation model from PyTorch format to ONNX format; Deploy the semantic segmentation model in ONNX format to the DLA.

[0006] In a possible implementation, the processing the sample image using the semantic segmentation model to obtain a prediction result includes: Using the feature extraction module to extract features from the sample image to obtain an image feature vector; Using the semantic extraction branch to perform semantic extraction on the image feature vector to obtain a semantic feature vector; Using the detail extraction branch and the semantic extraction branch to extract details from the image feature vector to obtain a detail feature vector; Using the boundary extraction branch and the semantic extraction branch to perform boundary extraction on the image feature vector to obtain a boundary feature vector; Using the fusion module to fuse the detail feature vector, the semantic feature vector and the boundary feature vector to obtain a fused feature vector; The fused feature vector is processed by using the upsampling segmentation head to obtain a prediction result.

[0007] In a possible implementation, the using the detail loss function, the three semantic loss functions, the boundary loss function and the two segmentation loss functions to perform loss calculation on the prediction result, the true category label and the true boundary label includes: Using the two auxiliary heads and the optimized feature pyramid to perform semantic prediction on the semantic feature vectors corresponding to each other, and using the semantic loss functions corresponding to the two auxiliary heads and the optimized feature pyramid to perform loss calculation on the corresponding semantic prediction results and the true category labels; Using the detail prediction head to perform detail prediction on the detail feature vector, and using the detail loss function to perform loss calculation on the detail prediction result and the true category label; Using the boundary prediction head to perform boundary prediction on the boundary feature vector, and using the boundary loss function to perform loss calculation on the boundary prediction result and the true boundary label; The fused feature vector is predicted using the upsampling segmentation head, and the loss of the prediction result and the true category label is calculated using a segmentation loss function; the fused feature vector is upsampled and predicted using the upsampling segmentation head, and the loss of the prediction result and the true category label is calculated using another segmentation loss function.

[0008] In a possible implementation, the using the fusion module to fuse the detail feature vector, the semantic feature vector, and the boundary feature vector to obtain a fused feature vector includes: Creating a fusion algorithm supported by the DLA in the fusion module; The detail feature vector, the semantic feature vector and the boundary feature vector are fused using the fusion algorithm to obtain a fused feature vector.

[0009] In a possible implementation, the extracting details from the image feature vector using the detail extraction branch and the semantic extraction branch to obtain the detail feature vector includes: When the detail extraction branch includes three detail modules and the semantic extraction branch includes three semantic modules, using the first detail module to perform detail extraction on the image feature vector to obtain a first detail feature, and using the first semantic module to perform semantic extraction on the image feature vector to obtain a first semantic feature; Using a second fusion algorithm supported by the DLA to fuse the first detail feature and the first semantic feature to obtain a first fused feature; using a second detail module to extract details from the first fused feature to obtain a second detail feature, and using a second semantic module to extract semantics from the first semantic feature to obtain a second semantic feature; The second detail feature and the second semantic feature are fused using a second fusion algorithm supported by the DLA to obtain a second fused feature; and the details of the second fused feature are extracted using a third detail module to obtain the detail feature vector.

[0010] In a possible implementation, the using the boundary extraction branch and the semantic extraction branch to extract the boundary of the image feature vector to obtain the boundary feature vector includes: When the boundary extraction branch includes three boundary modules and the semantic extraction branch includes three semantic modules, using the first boundary module to perform boundary extraction on the image feature vector to obtain a first boundary feature, and using the first semantic module to perform semantic extraction on the image feature vector to obtain a first semantic feature; Using a second boundary module to perform boundary extraction on the splicing of the first boundary feature and the first semantic feature to obtain a second boundary feature, and using a second semantic module to perform semantic extraction on the first semantic feature to obtain a second semantic feature; The third boundary module is used to perform boundary extraction on the concatenation of the second boundary feature and the second semantic feature to obtain the boundary feature vector.

[0011] In a possible implementation, the method further includes: Acquire a perception image to be recognized; Using the feature extraction module to extract features from the perceived image to obtain an image feature vector; Using the semantic extraction branch to perform semantic extraction on the image feature vector to obtain a semantic feature vector; Using the detail extraction branch and the semantic extraction branch to extract details from the image feature vector to obtain a detail feature vector; Using the boundary extraction branch and the semantic extraction branch to perform boundary extraction on the image feature vector to obtain a boundary feature vector; Using the fusion module to fuse the detail feature vector, the semantic feature vector and the boundary feature vector to obtain a fused feature vector; The fused feature vector is upsampled by the upsampling segmentation head and then predicted to obtain an image segmentation result.

[0012] According to a second aspect of the present application, a deployment device for a semantic segmentation model is provided, the device comprising: A creation module is used to create a semantic segmentation model suitable for a deep learning accelerator DLA, wherein the semantic segmentation model includes a feature extraction module, a detail extraction branch, a semantic extraction branch, a boundary extraction branch, a fusion module, and an upsampling segmentation head. The detail prediction head in the detail extraction branch corresponds to a detail loss function, the two auxiliary heads and the optimized feature pyramid in the semantic extraction branch correspond to a semantic loss function respectively, the boundary prediction head in the boundary extraction branch corresponds to a boundary loss function, and the upsampling segmentation head corresponds to two segmentation loss functions; An acquisition module is used to acquire a training set, wherein the training samples in the training set include sample images, true category labels and true boundary labels, and the true boundary labels are obtained by performing boundary extraction on the true category labels; A training module, for each training sample, using the semantic segmentation model to process the sample image to obtain a prediction result, using the detail loss function, three semantic loss functions, the boundary loss function and two segmentation loss functions to perform loss calculation on the prediction result, the true category label and the true boundary label, and training the model parameters of the semantic segmentation model according to the calculation results; The training module is further used to perform INT8 quantization training on the trained semantic segmentation model to convert the semantic segmentation model from the PyTorch format to the ONNX format; A deployment module is used to deploy the semantic segmentation model in ONNX format to the DLA.

[0013] According to a third aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the deployment method of the semantic segmentation model as described above.

[0014] According to a fourth aspect of the present application, a computer device is provided, wherein the computer device includes a deployment device for the above-mentioned semantic segmentation model.

[0015] The beneficial effects of the technical solution provided by this application include at least: By creating a semantic segmentation model suitable for DLA, the semantic segmentation model is trained using the training set, and then the trained semantic segmentation model is trained using INT8 quantization, so that the semantic segmentation model can be converted from PyTorch format to ONNX format. Finally, the semantic segmentation model in ONNX format is deployed on DLA, thereby realizing the migration of the semantic segmentation model from GPU to DLA. Without significantly sacrificing accuracy, the inference speed of the semantic segmentation model is optimized and the performance of the semantic segmentation model is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a structural schematic diagram of a semantic segmentation model according to some exemplary embodiments; Figure 2 is a schematic diagram of the structure of an upsampling segmentation head according to some exemplary embodiments; Figure 3is a flowchart of a method for deploying a semantic segmentation model provided by an embodiment of the present application; Figure 4 is a flowchart of a method for deploying a semantic segmentation model provided by an embodiment of the present application; Figure 5 is a flowchart of a method for using a semantic segmentation model provided by an embodiment of the present application; Figure 6 is a schematic diagram of an image segmentation result provided by an embodiment of the present application; Figure 7 is a schematic diagram of an image segmentation result provided by an embodiment of the present application; Figure 8 It is a structural block diagram of a deployment device for a semantic segmentation model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0019] Figure 1 A structural diagram of a semantic segmentation model suitable for a deep learning accelerator (DLA) is shown. The semantic segmentation model includes a feature extraction module 110, a detail extraction branch 120, a semantic extraction branch 130, a boundary extraction branch 140, a fusion module 150 and an upsampling segmentation head 160. When the semantic segmentation model also includes a preprocessing module, the input end of the preprocessing module is the input end of the semantic segmentation model, and the output end of the preprocessing module is connected to the input end of the feature extraction module 110; when the semantic segmentation model does not include a preprocessing module, the input end of the feature extraction module 110 is the input end of the semantic segmentation model, and the output end of the feature extraction module 110 is respectively connected to the input ends of the detail extraction branch 120, the semantic extraction branch 130, and the boundary extraction branch 140, and the output ends of the detail extraction branch 120, the semantic extraction branch 130, and the boundary extraction branch 140 are respectively connected to the input end of the fusion module 150, and the output end of the fusion module 150 is connected to the input end of the upsampling segmentation head 160, and the output end of the upsampling segmentation head 160 is the output end of the semantic segmentation model.

[0020] The preprocessing module is used to preprocess the input image. Specifically, the preprocessing module performs data enhancement and normalization operations such as scaling and rotation on the input image, and then inputs it into the feature extraction module at a resolution of 512×640.

[0021] The feature extraction module 110 is used to extract features from the image output by the preprocessing module to obtain an image feature vector.

[0022] The semantic extraction branch 130 includes semantic modules 1-3, an optimized feature pyramid, and three auxiliary heads. The semantic modules 1-3 are connected in series with the optimized feature pyramid. The semantic module 1 is connected to the first auxiliary head. The semantic loss function corresponding to the auxiliary head is Loss 16X ; Semantic module 2 is connected to the second auxiliary head, and the semantic loss function corresponding to the auxiliary head is Loss 32X ; The optimized feature pyramid is connected to the third auxiliary head, and the semantic loss function corresponding to the auxiliary head is Loss 64X , where 16X, 32X and 64X represent sampling multiples. The semantic extraction branch 130 is used to perform semantic extraction on the image feature vector output by the feature extraction module 110 to obtain a semantic feature vector.

[0023] The detail extraction branch 120 includes detail modules 1-3 and a detail prediction head. The detail modules 1-3 are connected in series with the detail prediction head. The detail prediction head corresponds to a detail loss function Loss p The detail extraction branch 120 is used to extract details from the image feature vector output by the feature extraction module 110 and the semantic feature vector output by the semantic extraction branch 130 to obtain a detail feature vector.

[0024] The boundary extraction branch 140 includes boundary modules 1-3 and a boundary prediction head. The boundary modules 1-3 are connected in series with the boundary prediction head. The boundary prediction head corresponds to a boundary loss function Loss d The boundary extraction branch 140 is used to perform boundary extraction on the image feature vector output by the feature extraction module 110 and the semantic feature vector output by the semantic extraction branch 130 to obtain a boundary feature vector.

[0025] The fusion module 150 is used to fuse the detail feature vector, the semantic feature vector and the boundary feature vector to obtain a fused feature vector.

[0026] The upsampling segmentation head 160 corresponds to two segmentation loss functions with different sampling rates, namely Loss 8X and Loss 4X , where 8X and 4X represent sampling multiples. The upsampling segmentation head 160 is used to perform prediction on the fused feature vector after upsampling, and output a prediction result, that is, an image segmentation result.

[0027] The structure of the up-sampling segmentation head 160 in this application is as follows Figure 2As shown, the upsampling segmentation head 160 first performs convolution (conv), normalization (norm), activation (act) and upsampling (upsample) operations on the input feature vector, and then performs normalization (norm), activation (act), convolution (conv), normalization (norm), activation (act), and convolution (conv) operations, and then outputs the image segmentation result.

[0028] like Figure 3 As shown, it shows a method flow chart of a method for deploying a semantic segmentation model provided by an embodiment of the present application, and the method for deploying a semantic segmentation model can be applied to a computer device. The method for deploying a semantic segmentation model may include: Step 301: Create a semantic segmentation model suitable for DLA.

[0029] The semantic segmentation model created in this embodiment is Figure 1 The semantic segmentation model shown is suitable for DLA.

[0030] Step 302: Obtain a training set. The training samples in the training set include sample images, true category labels, and true boundary labels. The true boundary labels are obtained by performing boundary extraction on the true category labels.

[0031] The training set includes multiple training samples, each of which includes a sample image, a true category label and a true boundary label. The true category label is a predetermined number of category labels, such as 16 category labels. For each category label, a boundary extraction can be performed to obtain a boundary label.

[0032] Step 303: For each training sample, the sample image is processed using the semantic segmentation model to obtain a prediction result, and the prediction result, the true category label and the true boundary label are calculated using a detail loss function, three semantic loss functions, a boundary loss function and two segmentation loss functions. The model parameters of the semantic segmentation model are trained according to the calculation results.

[0033] After obtaining the semantic segmentation model, the semantic segmentation model can be used to process each sample image, and multiple loss functions, true category labels, and true boundary labels are used to calculate the loss, and the model parameters of the semantic segmentation are trained through multiple iterations.

[0034] Step 304: Perform INT8 quantization training on the trained semantic segmentation model to convert the semantic segmentation model from the PyTorch format to the ONNX format.

[0035] Post-Training Quantization (PTQ) is a technique for quantizing a model after model training is completed. Its main purpose is to convert a trained floating-point model (usually FP32 precision) to a low-precision fixed-point model (such as INT8) to reduce the size of the model and speed up inference. PTQ does not require retraining the model, but only requires quantization on the existing model, saving time and computing resources. The quantized model can be deployed faster in resource-constrained environments, and the low-precision representation reduces storage space and computing overhead.

[0036] After the semantic segmentation model is trained for 85 epochs, the semantic segmentation model can be obtained under the PyTorch framework. The semantic segmentation model is trained with PTQ INT8 quantization and converted into a semantic segmentation model in ONNX format.

[0037] Step 305: deploy the semantic segmentation model in ONNX format to DLA.

[0038] The exported semantic segmentation models in ONNX format can all run on DLA with INT8 precision. Changing from running on GPU to running on DLA can not only release more computing resources for GPU running of other models, but also optimize the inference speed of the semantic segmentation model and improve the performance of the semantic segmentation model.

[0039] To summarize, the deployment method of the semantic segmentation model provided in the embodiment of the present application creates a semantic segmentation model suitable for DLA, trains the semantic segmentation model using a training set, and then performs INT8 quantization training on the trained semantic segmentation model so as to convert the semantic segmentation model from PyTorch format to ONNX format. Finally, the semantic segmentation model in ONNX format is deployed on DLA, thereby realizing the migration of the semantic segmentation model from GPU to DLA, optimizing the inference speed of the semantic segmentation model without significantly sacrificing accuracy, and improving the performance of the semantic segmentation model.

[0040] like Figure 4 As shown, it shows a flowchart of a method for deploying a semantic segmentation model provided by an embodiment of the present application, and the method for deploying a semantic segmentation model can be applied to a computer device. The method for deploying a semantic segmentation model may include: Step 401: Create a semantic segmentation model suitable for DLA.

[0041] The semantic segmentation model created in this embodiment is Figure 1 The semantic segmentation model shown is suitable for DLA.

[0042] Step 402, obtaining a training set, wherein the training samples in the training set include sample images, true category labels and true boundary labels, and the true boundary labels are obtained by performing boundary extraction on the true category labels.

[0043] The training set includes multiple training samples, each of which includes a sample image, a true category label and a true boundary label. The true category label is a predetermined number of category labels, such as 16 category labels. For each category label, a boundary extraction can be performed to obtain a boundary label.

[0044] Step 403: For each training sample, a feature extraction module is used to extract features from the sample image to obtain an image feature vector.

[0045] The feature extraction module extracts features from the sample image and obtains an 8X image feature vector, that is, an image feature vector with a resolution of 64×80.

[0046] Step 404: perform semantic extraction on the image feature vector using the semantic extraction branch to obtain a semantic feature vector.

[0047] Semantic modules 1-3 are semantic feature extraction modules with different convolution widths and depths. After the semantic extraction branch, the image feature vector is downsampled to 8X->16X->32X->64X. The 64X semantic context feature has a large receptive field and a strong ability to perceive global context. Finally, the 64X semantic context feature is processed by the optimized feature pyramid to obtain the semantic feature vector.

[0048] When extracting the semantic feature vector, the semantic module 1 outputs the extracted features to the first auxiliary head, the semantic module 2 outputs the extracted features to the second auxiliary head, and the optimized feature pyramid outputs the extracted semantic feature vector to the third auxiliary head.

[0049] The structures of the three auxiliary heads are the same, the difference is that the input scale is different. The formula for processing the input feature vector by the auxiliary head is: x=CONV(ConCat(x,DCN(x))) (1) Among them, x represents the feature vector, CONV represents convolution, ConCat represents concatenation, and DCN represents deformable convolution.

[0050] In the existing technology, the original feature pyramid uses adaptive average pooling (avgpooling) to aggregate features to 1×1 dimensions and then performs pyramid-shaped feature aggregation operations. However, DLA INT8 does not support the operation of this operator. Therefore, we directly use 1×1 convolution to replace the feature pyramid module, and call this module the optimized feature pyramid.

[0051] Step 405: Use the detail extraction branch and the semantic extraction branch to extract details from the image feature vector to obtain a detail feature vector.

[0052] Specifically, using the detail extraction branch and the semantic extraction branch to extract details from the image feature vector to obtain the detail feature vector may include: (1) When the detail extraction branch includes three detail modules and the semantic extraction branch includes three semantic modules, the first detail module is used to perform detail extraction on the image feature vector to obtain a first detail feature, and the first semantic module is used to perform semantic extraction on the image feature vector to obtain a first semantic feature.

[0053] (2) Using the second fusion algorithm supported by DLA to fuse the first detail feature and the first semantic feature, a first fused feature is obtained; using the second detail module to extract details from the first fused feature, a second detail feature is obtained; using the second semantic module to extract semantics from the first semantic feature, a second semantic feature is obtained.

[0054] (3) Using the second fusion algorithm supported by DLA to fuse the second detail feature and the second semantic feature to obtain a second fused feature; using the third detail module to extract details from the second fused feature to obtain a detail feature vector.

[0055] The first detail module is Figure 1 The detail module 1 in the second detail module is Figure 1 The detail module 2 in the third detail module is Figure 1 Details in Module 3.

[0056] Detail modules 1-3 are detail feature extraction modules with different convolution depths. In the detail extraction branch, the detail features of each layer are fused with the semantic features from the corresponding layer (F1) and then passed to the next layer for detail feature extraction. Fusion (F1) is the process of weighting the features from semantics and details. The specific formula is as follows: (2) (3) Among them, l and l-1 represent the lth layer and l-1th layer respectively. and They represent the semantic features and detail features from the l-1th layer, respectively. W1 is the operation of 1×1 convolution->upsample->1×1 convolution, and W2 is the operation of 1×1 convolution->3×3 convolution->1×1 convolution. b represents the offset. After feature extraction by the detail extraction branch, the detail information of the sample image is enriched, and no downsampling is performed at this time. Finally, an 8X detail feature vector is obtained.

[0057] Step 406: Use the boundary extraction branch and the semantic extraction branch to perform boundary extraction on the image feature vector to obtain a boundary feature vector.

[0058] Specifically, using the boundary extraction branch and the semantic extraction branch to extract the boundary of the image feature vector to obtain the boundary feature vector may include: (1) When the boundary extraction branch includes three boundary modules and the semantic extraction branch includes three semantic modules, the first boundary module is used to perform boundary extraction on the image feature vector to obtain a first boundary feature, and the first semantic module is used to perform semantic extraction on the image feature vector to obtain a first semantic feature.

[0059] (2) Using the second boundary module to perform boundary extraction on the concatenation of the first boundary feature and the first semantic feature to obtain a second boundary feature, and using the second semantic module to perform semantic extraction on the first semantic feature to obtain a second semantic feature.

[0060] (3) Use the third boundary module to extract the boundary by concatenating the second boundary feature and the second semantic feature to obtain a boundary feature vector.

[0061] The first boundary module is Figure 1 The boundary module 1 in the second boundary module is Figure 1 The third boundary module is Figure 1 Boundary module 3 in.

[0062] The boundary extraction branch obtains the 8X boundary feature vector through ConCat{semantic, boundary} features.

[0063] Step 407: Use a fusion module to fuse the detail feature vector, the semantic feature vector, and the boundary feature vector to obtain a fused feature vector.

[0064] Specifically, using a fusion module to fuse detail feature vectors, semantic feature vectors and boundary feature vectors to obtain a feature vector may include: creating a fusion algorithm supported by DLA in the fusion module; using the fusion algorithm to fuse the detail feature vectors, semantic feature vectors and boundary feature vectors to obtain a fused feature vector.

[0065] The formula of the fusion algorithm is: f out =F2(f p , f i , f d ) (4) F2(f p , f i , f d )=W3×f p +W4×fi +W5×f d +c (5) Among them, f p Represents the detail feature vector of 8X, f i represents the semantic feature vector of 8X, f d represents the boundary feature vector of 8X, W3, W4 and W5 represent a series of operations, and c represents the offset.

[0066] Step 408: Use an upsampling segmentation head to process the fused feature vector to obtain a prediction result.

[0067] The original segmentation head outputs the original scale without sampling, and the final output result is an 8X feature vector. However, the 8X feature vector does not have fine boundaries and clear details in the distance. To optimize the effect, we added a layer of convolution and 2x upsampling before the original segmentation head, upsampled the 8X feature vector to 4x before output supervision, and optimized the details in use.

[0068] The fused feature vector obtained by the fusion module will be output to the upsampling segmentation head, which will output the final prediction result.

[0069] Step 409 , using a detail loss function, three semantic loss functions, a boundary loss function and two segmentation loss functions, loss calculations are performed on the prediction results, the true category labels and the true boundary labels, and model parameters of the semantic segmentation model are trained according to the calculation results.

[0070] Specifically, the loss calculation of the prediction results, the true category labels and the true boundary labels is performed using the detail loss function, three semantic loss functions, the boundary loss function and two segmentation loss functions, which may include: (1) Use the two auxiliary heads and the optimized feature pyramid to perform semantic prediction on the corresponding semantic feature vectors, and use the semantic loss functions corresponding to the two auxiliary heads and the optimized feature pyramid to calculate the loss between the corresponding semantic prediction results and the true category labels.

[0071] (2) Use the detail prediction head to predict the detail feature vector, and use the detail loss function to calculate the loss between the detail prediction result and the true category label.

[0072] (3) Use the boundary prediction head to predict the boundary feature vector, and use the boundary loss function to calculate the loss between the boundary prediction result and the true boundary label.

[0073] (4) Use the upsampling segmentation head to predict the fused feature vector, and use a segmentation loss function to calculate the loss between the prediction result and the true category label; use the upsampling segmentation head to upsample the fused feature vector and make a prediction, and use another segmentation loss function to calculate the loss between the prediction result and the true category label.

[0074] These results need to be calculated pixel by pixel with the true category labels and true boundary labels to update the model gradients and constrain parameter optimization directions.

[0075] Among them, the segmentation loss function Loss corresponding to the upsampling segmentation head (including Loss 8X and Loss 4X ) is OhemCrossEntropy, the detail loss function corresponding to the detail prediction head Loss p It is CrossEntropyLoss, the boundary loss function Loss corresponding to the boundary prediction head d is BoundaryLoss, the semantic loss function Loss corresponding to the auxiliary head (including Loss 16X 、Loss 32X 、Loss 64X ) is CrossEntropyLoss.

[0076] Step 410: Perform INT8 quantization training on the trained semantic segmentation model to convert the semantic segmentation model from the PyTorch format to the ONNX format.

[0077] PTQ is a technology that quantizes models after model training is completed. Its main purpose is to convert a trained floating-point model (usually FP32 precision) to a low-precision fixed-point model (such as INT8) to reduce the size of the model and speed up inference. PTQ does not require retraining the model, but only requires quantization on the existing model, saving time and computing resources. The quantized model can be deployed faster in resource-constrained environments, and the low-precision representation reduces storage space and computing overhead.

[0078] After the semantic segmentation model is trained for 85 epochs, the semantic segmentation model can be obtained under the PyTorch framework. The semantic segmentation model is trained with PTQ INT8 quantization and converted into a semantic segmentation model in ONNX format.

[0079] Step 411, deploy the semantic segmentation model in ONNX format to DLA.

[0080] The exported semantic segmentation models in ONNX format can all run on DLA with INT8 precision. Changing from running on GPU to running on DLA can not only release more computing resources for GPU running of other models, but also optimize the inference speed of the semantic segmentation model and improve the performance of the semantic segmentation model.

[0081] To summarize, the deployment method of the semantic segmentation model provided in the embodiment of the present application creates a semantic segmentation model suitable for DLA, trains the semantic segmentation model using a training set, and then performs INT8 quantization training on the trained semantic segmentation model so as to convert the semantic segmentation model from PyTorch format to ONNX format. Finally, the semantic segmentation model in ONNX format is deployed on DLA, thereby realizing the migration of the semantic segmentation model from GPU to DLA, optimizing the inference speed of the semantic segmentation model without significantly sacrificing accuracy, and improving the performance of the semantic segmentation model.

[0082] After the semantic segmentation model is deployed on DLA, the semantic segmentation model can be used to perform semantic segmentation on the perception images collected by the vehicle during driving. The specific process is as follows: Figure 5 As shown: Step 501: Obtain a perceived image to be recognized.

[0083] Step 502: Use a feature extraction module to extract features from the perceived image to obtain an image feature vector.

[0084] Step 503: perform semantic extraction on the image feature vector using the semantic extraction branch to obtain a semantic feature vector.

[0085] Step 504: extract details from the image feature vector using the detail extraction branch and the semantic extraction branch to obtain a detail feature vector.

[0086] Step 505: Use the boundary extraction branch and the semantic extraction branch to extract the boundary of the image feature vector to obtain a boundary feature vector.

[0087] Step 506: Use a fusion module to fuse the detail feature vector, the semantic feature vector, and the boundary feature vector to obtain a fused feature vector.

[0088] Step 507: Use an upsampling segmentation head to upsample the fused feature vector and then perform prediction to obtain an image segmentation result.

[0089] It should be noted that in the process of using the semantic segmentation model, the detail prediction head, three auxiliary heads, and boundary prediction head do not work.

[0090] Figure 6 and Figure 7The image segmentation results obtained using the semantic segmentation model are shown. When BatchSize=5, the semantic segmentation model running on the DLA platform has an inference speed of about 75ms, achieving a balance between speed and accuracy.

[0091] like Figure 8 As shown, it shows a structural block diagram of a deployment device of a semantic segmentation model provided by an embodiment of the present application, and the deployment device of the semantic segmentation model can be applied to a computer device. The deployment device of the semantic segmentation model may include:

[0092] A model creation module 810 is used to create a semantic segmentation model suitable for DLA, wherein the semantic segmentation model includes a feature extraction module, a detail extraction branch, a semantic extraction branch, a boundary extraction branch, a fusion module, and an upsampling segmentation head. The detail prediction head in the detail extraction branch corresponds to a detail loss function, the two auxiliary heads and the optimized feature pyramid in the semantic extraction branch correspond to a semantic loss function respectively, the boundary prediction head in the boundary extraction branch corresponds to a boundary loss function, and the upsampling segmentation head corresponds to two segmentation loss functions; A training set acquisition module 820 is used to acquire a training set, wherein the training samples in the training set include sample images, true category labels and true boundary labels, and the true boundary labels are obtained by performing boundary extraction on the true category labels; A model training module 830 is used to process the sample image using the semantic segmentation model for each training sample to obtain a prediction result, calculate the loss of the prediction result, the true category label and the true boundary label using a detail loss function, three semantic loss functions, a boundary loss function and two segmentation loss functions, and train the model parameters of the semantic segmentation model according to the calculation results; The model training module 830 is further used to perform INT8 quantization training on the trained semantic segmentation model to convert the semantic segmentation model from the PyTorch format to the ONNX format; The model deployment module 840 is used to deploy the semantic segmentation model in ONNX format to DLA.

[0093] In an optional embodiment, the model training module 830 is further used to: Use the feature extraction module to extract features from the sample image and obtain the image feature vector; The semantic extraction branch is used to perform semantic extraction on the image feature vector to obtain a semantic feature vector; Utilize the detail extraction branch and the semantic extraction branch to extract details from the image feature vector to obtain a detail feature vector; The boundary extraction branch and the semantic extraction branch are used to extract the boundary of the image feature vector to obtain the boundary feature vector; The fusion module is used to fuse the detail feature vector, the semantic feature vector and the boundary feature vector to obtain a fused feature vector; The fused feature vector is processed using the upsampling segmentation head to obtain the prediction result.

[0094] In an optional embodiment, the model training module 830 is further used to: Use two auxiliary heads and optimized feature pyramids to perform semantic prediction on the corresponding semantic feature vectors, and use the semantic loss functions corresponding to the two auxiliary heads and optimized feature pyramids to calculate the loss of the corresponding semantic prediction results and the true category labels; Use the detail prediction head to predict the detail feature vector, and use the detail loss function to calculate the loss between the detail prediction result and the true category label; The boundary prediction head is used to predict the boundary feature vector, and the boundary loss function is used to calculate the loss between the boundary prediction result and the actual boundary label; The fused feature vector is predicted using the upsampling segmentation head, and the loss between the prediction result and the true category label is calculated using a segmentation loss function; the fused feature vector is upsampled and predicted using the upsampling segmentation head, and the loss between the prediction result and the true category label is calculated using another segmentation loss function.

[0095] In an optional embodiment, the model training module 830 is further used to: Create a DLA-supported fusion algorithm in the fusion module; The detail feature vector, semantic feature vector and boundary feature vector are fused using a fusion algorithm to obtain a fused feature vector.

[0096] In an optional embodiment, the model training module 830 is further used to: When the detail extraction branch includes three detail modules and the semantic extraction branch includes three semantic modules, the first detail module is used to perform detail extraction on the image feature vector to obtain a first detail feature, and the first semantic module is used to perform semantic extraction on the image feature vector to obtain a first semantic feature; The first detail feature and the first semantic feature are fused by using the second fusion algorithm supported by DLA to obtain a first fused feature; the second detail module is used to extract details from the first fused feature to obtain a second detail feature, and the second semantic module is used to extract semantics from the first semantic feature to obtain a second semantic feature; The second detail feature and the second semantic feature are fused using the second fusion algorithm supported by DLA to obtain a second fused feature; the third detail module is used to extract details from the second fused feature to obtain a detail feature vector.

[0097] In an optional embodiment, the model training module 830 is further used to: When the boundary extraction branch includes three boundary modules and the semantic extraction branch includes three semantic modules, the first boundary module is used to perform boundary extraction on the image feature vector to obtain a first boundary feature, and the first semantic module is used to perform semantic extraction on the image feature vector to obtain a first semantic feature; Using the second boundary module to perform boundary extraction on the splicing of the first boundary feature and the first semantic feature to obtain a second boundary feature, and using the second semantic module to perform semantic extraction on the first semantic feature to obtain a second semantic feature; The third boundary module is used to extract the boundary by concatenating the second boundary feature and the second semantic feature to obtain a boundary feature vector.

[0098] In an optional embodiment, the device further comprises: An image acquisition module, used for acquiring a perception image to be recognized; A feature extraction module is used to extract features from the perceived image using the feature extraction module to obtain an image feature vector; The feature extraction module is also used to perform semantic extraction on the image feature vector using the semantic extraction branch to obtain a semantic feature vector; The feature extraction module is further used to extract details from the image feature vector using the detail extraction branch and the semantic extraction branch to obtain a detail feature vector; The feature extraction module is further used to extract the boundary of the image feature vector using the boundary extraction branch and the semantic extraction branch to obtain a boundary feature vector; The feature extraction module is also used to fuse the detail feature vector, the semantic feature vector and the boundary feature vector using the fusion module to obtain a fused feature vector; The result prediction module is used to use the upsampling segmentation head to upsample the fused feature vector and then perform prediction to obtain the image segmentation result.

[0099] To summarize, the semantic segmentation model deployment device provided in the embodiment of the present application creates a semantic segmentation model suitable for DLA, trains the semantic segmentation model using a training set, and then performs INT8 quantization training on the trained semantic segmentation model so as to convert the semantic segmentation model from PyTorch format to ONNX format. Finally, the semantic segmentation model in ONNX format is deployed on DLA, thereby realizing the migration of the semantic segmentation model from GPU to DLA, optimizing the inference speed of the semantic segmentation model without significantly sacrificing accuracy, and improving the performance of the semantic segmentation model.

[0100] An embodiment of the present application provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the deployment method of the semantic segmentation model as described above.

[0101] An embodiment of the present application provides a computer device, which includes a deployment device for any of the above-mentioned semantic segmentation models.

[0102] It should be noted that: the deployment device of the semantic segmentation model provided in the above embodiment only uses the division of the above functional modules as an example when deploying the semantic segmentation model. In actual applications, the above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the deployment device of the semantic segmentation model is divided into different functional modules to complete all or part of the functions described above. In addition, the deployment device of the semantic segmentation model provided in the above embodiment and the deployment method embodiment of the semantic segmentation model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0103] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0104] The above description is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A method for deploying a semantic segmentation model, characterized in that: The method comprises: Creating a semantic segmentation model suitable for a deep learning accelerator DLA, the semantic segmentation model comprising a feature extraction module, a detail extraction branch, a semantic extraction branch, a boundary extraction branch, a fusion module, and an upsampling segmentation head, wherein the detail prediction head in the detail extraction branch corresponds to a detail loss function, the two auxiliary heads and the optimized feature pyramid in the semantic extraction branch correspond to a semantic loss function respectively, the boundary prediction head in the boundary extraction branch corresponds to a boundary loss function, and the upsampling segmentation head corresponds to two segmentation loss functions with different sampling rates; Obtaining a training set, wherein the training samples in the training set include sample images, true category labels, and true boundary labels, wherein the true boundary labels are obtained by performing boundary extraction on the true category labels; For each training sample, the sample image is processed using the semantic segmentation model to obtain a prediction result, and the prediction result, the true category label and the true boundary label are calculated using the detail loss function, three semantic loss functions, the boundary loss function and two segmentation loss functions, and the model parameters of the semantic segmentation model are trained according to the calculation results; Perform INT8 quantization training on the trained semantic segmentation model to convert the semantic segmentation model from PyTorch format to ONNX format; Deploy the semantic segmentation model in ONNX format to the DLA.

2. The method for deploying a semantic segmentation model according to claim 1, characterized in that: The using the semantic segmentation model to process the sample image to obtain a prediction result includes: Using the feature extraction module to extract features from the sample image to obtain an image feature vector; Using the semantic extraction branch to perform semantic extraction on the image feature vector to obtain a semantic feature vector; Using the detail extraction branch and the semantic extraction branch to extract details from the image feature vector to obtain a detail feature vector; Using the boundary extraction branch and the semantic extraction branch to perform boundary extraction on the image feature vector to obtain a boundary feature vector; Using the fusion module to fuse the detail feature vector, the semantic feature vector and the boundary feature vector to obtain a fused feature vector; The fused feature vector is processed by using the upsampling segmentation head to obtain a prediction result.

3. The method for deploying a semantic segmentation model according to claim 2, characterized in that: The using the detail loss function, the three semantic loss functions, the boundary loss function and the two segmentation loss functions to perform loss calculation on the prediction result, the true category label and the true boundary label includes: Using the two auxiliary heads and the optimized feature pyramid to perform semantic prediction on the semantic feature vectors corresponding to each other, and using the semantic loss functions corresponding to the two auxiliary heads and the optimized feature pyramid to perform loss calculation on the corresponding semantic prediction results and the true category labels; Using the detail prediction head to perform detail prediction on the detail feature vector, and using the detail loss function to perform loss calculation on the detail prediction result and the true category label; Using the boundary prediction head to perform boundary prediction on the boundary feature vector, and using the boundary loss function to perform loss calculation on the boundary prediction result and the true boundary label; The fused feature vector is predicted using the upsampling segmentation head, and the loss of the prediction result and the true category label is calculated using a segmentation loss function; the fused feature vector is upsampled and predicted using the upsampling segmentation head, and the loss of the prediction result and the true category label is calculated using another segmentation loss function.

4. The method for deploying a semantic segmentation model according to claim 2, characterized in that: The step of fusing the detail feature vector, the semantic feature vector and the boundary feature vector using the fusion module to obtain a fused feature vector includes: Creating a fusion algorithm supported by the DLA in the fusion module; The detail feature vector, the semantic feature vector and the boundary feature vector are fused using the fusion algorithm to obtain a fused feature vector.

5. The method for deploying a semantic segmentation model according to claim 2, characterized in that: The step of extracting details from the image feature vector using the detail extraction branch and the semantic extraction branch to obtain a detail feature vector includes: When the detail extraction branch includes three detail modules and the semantic extraction branch includes three semantic modules, using the first detail module to perform detail extraction on the image feature vector to obtain a first detail feature, and using the first semantic module to perform semantic extraction on the image feature vector to obtain a first semantic feature; Using a second fusion algorithm supported by the DLA to fuse the first detail feature and the first semantic feature to obtain a first fused feature; using a second detail module to extract details from the first fused feature to obtain a second detail feature, and using a second semantic module to extract semantics from the first semantic feature to obtain a second semantic feature; The second detail feature and the second semantic feature are fused using a second fusion algorithm supported by the DLA to obtain a second fused feature; and the details of the second fused feature are extracted using a third detail module to obtain the detail feature vector.

6. The method for deploying a semantic segmentation model according to claim 2, characterized in that: The step of performing boundary extraction on the image feature vector by using the boundary extraction branch and the semantic extraction branch to obtain a boundary feature vector includes: When the boundary extraction branch includes three boundary modules and the semantic extraction branch includes three semantic modules, using the first boundary module to perform boundary extraction on the image feature vector to obtain a first boundary feature, and using the first semantic module to perform semantic extraction on the image feature vector to obtain a first semantic feature; Using a second boundary module to perform boundary extraction on the splicing of the first boundary feature and the first semantic feature to obtain a second boundary feature, and using a second semantic module to perform semantic extraction on the first semantic feature to obtain a second semantic feature; The third boundary module is used to perform boundary extraction on the concatenation of the second boundary feature and the second semantic feature to obtain the boundary feature vector.

7. The method for deploying a semantic segmentation model according to claim 1, characterized in that: The method further comprises: Acquire a perception image to be recognized; Using the feature extraction module to extract features from the perceived image to obtain an image feature vector; Using the semantic extraction branch to perform semantic extraction on the image feature vector to obtain a semantic feature vector; Using the detail extraction branch and the semantic extraction branch to extract details from the image feature vector to obtain a detail feature vector; Using the boundary extraction branch and the semantic extraction branch to perform boundary extraction on the image feature vector to obtain a boundary feature vector; Using the fusion module to fuse the detail feature vector, the semantic feature vector and the boundary feature vector to obtain a fused feature vector; The fused feature vector is upsampled by the upsampling segmentation head and then predicted to obtain an image segmentation result.

8. A deployment device for a semantic segmentation model, characterized in that: The device comprises: A creation module is used to create a semantic segmentation model suitable for a deep learning accelerator DLA, wherein the semantic segmentation model includes a feature extraction module, a detail extraction branch, a semantic extraction branch, a boundary extraction branch, a fusion module, and an upsampling segmentation head. The detail prediction head in the detail extraction branch corresponds to a detail loss function, the two auxiliary heads and the optimized feature pyramid in the semantic extraction branch correspond to a semantic loss function respectively, the boundary prediction head in the boundary extraction branch corresponds to a boundary loss function, and the upsampling segmentation head corresponds to two segmentation loss functions; An acquisition module is used to acquire a training set, wherein the training samples in the training set include sample images, true category labels and true boundary labels, and the true boundary labels are obtained by performing boundary extraction on the true category labels; A training module, for each training sample, using the semantic segmentation model to process the sample image to obtain a prediction result, using the detail loss function, three semantic loss functions, the boundary loss function and two segmentation loss functions to perform loss calculation on the prediction result, the true category label and the true boundary label, and training the model parameters of the semantic segmentation model according to the calculation results; The training module is further used to perform INT8 quantization training on the trained semantic segmentation model to convert the semantic segmentation model from the PyTorch format to the ONNX format; A deployment module is used to deploy the semantic segmentation model in ONNX format to the DLA.

9. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the deployment method of the semantic segmentation model as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device includes: a deployment device for the semantic segmentation model described in claim 8.

Images