Semantic segmentation method, image processing device, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a semantic segmentation method, an image processing device and a computer readable storage medium, which are used for performing semantic segmentation on an original image while improving segmentation accuracy. The method comprises the following steps: after obtaining the original image, inputting the original image into a multi-scale feature extraction module of a semantic segmentation model to obtain a first feature map of N-level scales of the original image output by the multi-scale feature extraction module, wherein N is an integer greater than or equal to 2; the semantic segmentation model is a pre-trained model; inputting the first feature map of the N-level scales into a target fusion module of the semantic segmentation model respectively; performing at least one fusion processing on the first feature map of the N-level scales by the target fusion module to obtain a target fusion feature map of the N-level scales output by the target fusion module; and generating a semantic segmentation image corresponding to the original image according to the target fusion feature map of the N-level scales.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of image processing, and more specifically, to a semantic segmentation method, an image processing device, and a computer readable storage medium. BACKGROUND

[0002] The current automobile industry is in an era of change. Autonomous driving is a function of vehicles to realize driving without driver operation. Related technologies are developing and applying like wildfire, with various development paths, such as single vehicle intelligence, vehicle-road cooperation, and networked cloud control. Autonomous driving technology senses the vehicle and the surrounding environment in real time through sensors, makes planning and decision-making through an intelligent system, and finally performs driving operations through a control system. Although there are many researchers in the related technology industry, there are still many deficiencies in the segmentation field of autonomous driving, such as too slow recognition of moving objects, too low accuracy, and inaccurate prediction of static objects such as lane lines, which may lead to serious traffic accidents and endanger people's personal and property safety. Therefore, the comprehensiveness and accuracy of image feature extraction in autonomous driving image semantic segmentation play a key role in the entire automobile industry.

[0003] The existing semantic segmentation method is a local pyramid occupancy network (PON) semantic segmentation method. The local pyramid occupancy network (PON) takes a residual module as the backbone feature extraction network, and in the deep convolutional neural network architecture, contains a Transformer module running at multiple image scales. The specific process is as follows: first, an original image is obtained, then the original image is input into a multi-scale feature extraction module to obtain a feature map of the original image at multiple scales output by the multi-scale feature extraction module, the multi-scale feature extraction module can be a residual module, etc., the residual module can be resnet50, etc., then the feature maps at multiple scales are input into the Transformer module to obtain a feature map output by the Transformer module, and finally a semantic segmentation image of the original image is generated according to the feature map output by the Transformer module.

[0004] However, the Transformer module has poor comprehensiveness and accuracy in image feature extraction, resulting in low segmentation accuracy, i.e., the intersection over union (IOU) of the semantic segmentation image output by the model and the semantic segmentation image of the label is low. SUMMARY

[0005] Embodiments of the present application provide a semantic segmentation method, an image processing device, and a computer readable storage medium, which are used to perform semantic segmentation on an original image while improving the intersection over union of the semantic segmentation image output by the model and the semantic segmentation image of the label.

[0006] In a first aspect, an embodiment of the present application provides a semantic segmentation method, comprising:

[0007] After obtaining an original image, input the original image into a multi-scale feature extraction module of a semantic segmentation model to obtain a first feature map of N-level scales of the original image output by the multi-scale feature extraction module; N is an integer greater than or equal to 2; the semantic segmentation model is a pre-trained model;

[0008] Input the first feature map of N-level scales into a target fusion module of the semantic segmentation model respectively, and obtain a target fusion feature map of N-level scales output by the target fusion module after at least one fusion processing of the first feature map of N-level scales by the target fusion module.

[0009] Generate a semantic segmentation image corresponding to the original image according to the target fusion feature map of N-level scales.

[0010] Optionally, before inputting the original image into the multi-scale feature extraction module of the semantic segmentation model, the method further comprises:

[0011] Preprocess the original image to obtain a preprocessed original image;

[0012] The inputting of the original image into the multi-scale feature extraction module of the semantic segmentation model comprises:

[0013] Input the preprocessed original image into the multi-scale feature extraction module.

[0014] Optionally, the target fusion module comprises a first fusion module and a second fusion module.

[0015] After the at least one fusion processing of the first feature map of N-level scales by the target fusion module, the target fusion module outputs a target fusion feature map of N-level scales, comprising:

[0016] The first fusion module performs first fusion processing on the first feature map of N-level scales to obtain a first fusion feature map of N-level scales output by the first fusion module;

[0017] The second fusion module performs second fusion processing on the first fusion feature map of N-level scales to obtain a second fusion feature map of N-level scales output by the second fusion module; wherein the second fusion feature map of N-level scales is the target fusion feature map of N-level scales.

[0018] Optionally, the first fusion module performs first fusion processing on the first feature maps of the N levels of scales to obtain first fusion feature maps of the N levels of scales output by the first fusion module, including:

[0019] For the first fusion feature maps of the n-1 level of scales, the first feature maps of the n level of scales are input into an up-sampling module of the semantic segmentation model, and the up-sampling module performs up-sampling processing to obtain up-sampling feature maps of the n level of scales; wherein 2≤n≤N;

[0020] The first fusion module performs first fusion processing on the up-sampling feature maps of the n level of scales and the first feature maps of the n-1 level of scales to obtain first fusion feature maps of the n-1 level of scales output by the first fusion module;

[0021] For the first fusion feature maps of the n level of scales, the first feature maps of the n level of scales are taken as the first fusion feature maps of the n level of scales.

[0022] Optionally, the second fusion module performs second fusion processing on the first fusion feature maps of the N levels of scales to obtain second fusion feature maps of the N levels of scales output by the second fusion module, including:

[0023] For the second fusion feature maps of the 1 level of scales, the first fusion feature maps of the 1 level of scales are taken as the second fusion feature maps of the 1 level of scales;

[0024] For the second fusion feature maps of the n level of scales, the first feature maps of the n-1 level of scales are input into a down-sampling module of the semantic segmentation model, and the down-sampling module performs down-sampling processing to obtain down-sampling feature maps of the n-1 level of scales; wherein 2≤n≤N;

[0025] The second fusion module performs second fusion processing on the down-sampling feature maps of the n-1 level of scales and the first fusion feature maps of the n level of scales to obtain second fusion feature maps of the n level of scales output by the second fusion module;

[0026] Optionally, the N level of scales target fusion feature maps are input into a global convolution module of the semantic segmentation model to obtain second feature maps of the N level of scales output by the global convolution module;

[0027] The N level of scales target fusion feature maps are input into a global convolution module of the semantic segmentation model to obtain second feature maps of the N level of scales output by the global convolution module;

[0028] The N level of scales target fusion feature maps are input into a global convolution module of the semantic segmentation model to obtain second feature maps of the N level of scales output by the global convolution module;

[0029] Optionally, the generating the semantic segmentation image corresponding to the original image according to the second feature map of the N-level scale comprises:

[0030] inputting the second feature map of the N-level scale into an edge refinement module of the semantic segmentation model to obtain a third feature map of the N-level scale output by the edge refinement module;

[0031] generating the semantic segmentation image corresponding to the original image according to the third feature map.

[0032] Optionally, before the inputting the original image into the multi-scale feature extraction module of the semantic segmentation model, the method further comprises:

[0033] obtaining original image samples, wherein each original image sample is labeled with a semantic segmentation image;

[0034] inputting the original image samples into a semantic segmentation model to obtain predicted semantic segmentation images corresponding to the original image samples output by the semantic segmentation model;

[0035] calculating a loss between the predicted semantic segmentation images and the labeled semantic segmentation images according to a regression loss function, and obtaining a trained semantic segmentation model when the loss meets a convergence condition.

[0036] In a second aspect, an image processing device is provided, comprising:

[0037] a central processor, a memory, an input-output interface, a wired or wireless network interface, and a power supply;

[0038] the memory is a transitory storage memory or a persistent storage memory;

[0039] the central processor is configured to communicate with the memory and execute instruction operations in the memory to perform the aforementioned semantic segmentation method.

[0040] In a third aspect, a computer-readable storage medium is provided, which comprises instructions, when the instructions are run on a computer, causing the computer to execute the aforementioned method.

[0041] In a fourth aspect, a computer program product comprising instructions is provided, when the computer program product is run on a computer, causing the computer to execute the aforementioned semantic segmentation method.

[0042] From the above technical solution, the embodiments of the present application have the following advantages: after obtaining the first feature map of N-level scales of the original image, the first feature map of N-level scales is respectively input into the target fusion module of the semantic segmentation model, and the target fusion module performs at least one fusion processing on the first feature map of N-level scales, thereby improving the comprehensiveness and accuracy of image feature extraction and improving the intersection over union of the semantic segmentation image output by the model and the semantic segmentation image of the label. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flowchart of a semantic segmentation method disclosed by the embodiments of the present application is shown in the figure.

[0044] Figure 2 A flowchart of a method for inputting an original image into a semantic segmentation model to obtain a semantic segmentation image is shown in the figure.

[0045] Figure 3 A structural diagram of a global convolution module disclosed by the embodiments of the present application is shown in the figure.

[0046] Figure 4 A structural diagram of an edge refinement module disclosed by the embodiments of the present application is shown in the figure.

[0047] Figure 5 A structural diagram of another image processing device disclosed by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0048] The embodiments of the present application provide a semantic segmentation method, an image processing device, and a computer readable storage medium, which are used to perform semantic segmentation on an original image while improving the intersection over union of the semantic segmentation image output by the model and the semantic segmentation image of the label.

[0049] Please refer to Figure 1 , Figure 1 A flowchart of a semantic segmentation method disclosed by the embodiments of the present application is shown in the figure, and the method comprises:

[0050] 101、After obtaining an original image, the original image is input into a multi-scale feature extraction module of a semantic segmentation model to obtain a first feature map of N-level scales of the original image output by the multi-scale feature extraction module; N is an integer greater than or equal to 2; the semantic segmentation model is a pre-trained model.

[0051] In this embodiment, when performing semantic segmentation, an original image can be obtained. After obtaining the original image, the original image can be input into a multi-scale feature extraction module of a semantic segmentation model to obtain a first feature map of N-level scales of the original image output by the multi-scale feature extraction module; N is an integer greater than or equal to 2; the semantic segmentation model is a pre-trained model.

[0052] 102. Input the first feature map at the N-level scale into the target fusion module of the semantic segmentation model. The target fusion module performs at least one fusion process on the first feature map at the N-level scale to obtain the target fusion feature map at the N-level scale output by the target fusion module.

[0053] After obtaining the first feature map at scale N of the original image output by the multi-scale feature extraction module, the first feature map at scale N can be input into the target fusion module of the semantic segmentation model. The target fusion module performs at least one fusion process on the first feature map at scale N to obtain the target fusion feature map at scale N output by the target fusion module. It can be understood that the method of performing at least one fusion process can be to first perform a first fusion process on the first feature map at scale N to obtain the first fused feature map at scale N, and then perform a second fusion process on the first fused feature map at scale N to obtain the second fused feature map at scale N; where the second fused feature map at scale N is the target fusion feature map at scale N. Other reasonable methods for performing at least one fusion process can also be used, and the specific method is not limited here.

[0054] 103. Generate the semantic segmentation image corresponding to the original image based on the target fusion feature map at the N-level scale.

[0055] After obtaining the N-level scale target fusion feature map output by the target fusion module, a semantic segmentation image corresponding to the original image can be generated based on the N-level scale target fusion feature map.

[0056] In this embodiment, after obtaining the first feature map at the N-level scale of the original image, the first feature map at the N-level scale can be input into the target fusion module of the semantic segmentation model. The target fusion module performs at least one fusion process on the first feature map at the N-level scale, which improves the comprehensiveness and accuracy of image feature extraction and increases the intersection-union ratio between the semantic segmentation image output by the model and the semantic segmentation image of the label.

[0057] In this embodiment of the application, there are multiple methods for obtaining the target fusion feature map at the N-level scale output by the target fusion module after performing at least one fusion process on the first feature map at the N-level scale by the target fusion module. Figure 1 The semantic segmentation methods shown below include one of them.

[0058] In this embodiment, the original image can be obtained during semantic segmentation. Specifically, semantic segmentation refers to semantically segmenting many objects in the original image, such as drivable areas, lane lines, buses, cars, pedestrians, and obstacles, into semantic segmentation maps (bird's-eye view BEV). The original image can be obtained from a large-scale public dataset for autonomous driving (Nuscenes dataset). The Nuscenes dataset can be collected from 1000 scenes in different cities using a data collection vehicle equipped with comprehensive sensors, including 6 cameras (CAM), 1 LiDAR, 5 millimeter-wave radars (RADAR), an IMU, and GPS. Other reasonable methods for obtaining the original image can also be used, and specific methods are not limited here.

[0059] After obtaining the original image, it can be preprocessed to obtain a preprocessed original image. Specifically, preprocessing methods can include normalization, scaling, etc. Normalization refers to converting the pixel values ​​of the original image to between 0 and 1, while scaling refers to using the resize() function to scale the height and width of the input image. It is important to understand that preprocessing the original image can reduce the network model's dependence on certain features, thereby improving the model's generalization and robustness.

[0060] After obtaining the preprocessed original image, it can be input into the multi-scale feature extraction module of the semantic segmentation model to obtain the first feature map of the original image at scale N, where N is an integer greater than or equal to 2. The semantic segmentation model is a pre-trained model. It can be understood that the pre-trained semantic segmentation model can be an improved FPN (Feature Pyramid Occupation Network) semantic segmentation model, and the multi-scale feature extraction module can be a residual module (ResNet50 module). For example, the size of the preprocessed original image can be 3*196*200. Inputting this 3*196*200 preprocessed original image into the ResNet50 module will yield the first feature map of the original image at scale N, as output by the ResNet50 module. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart illustrating a method for obtaining a semantically segmented image by inputting an original image into a semantic segmentation model according to an embodiment of this application. Figure 2In this case, N is 5. The first feature map at the first scale can be a 1 / 8 scale first feature map, the first feature map at the second scale can be a 1 / 16 scale first feature map, the first feature map at the third scale can be a 1 / 32 scale first feature map, the first feature map at the fourth scale can be a 1 / 64 scale first feature map, and the first feature map at the fifth scale can be a 1 / 128 scale first feature map. It is worth mentioning that the lower the scale level, the larger the scale size, and the higher the scale level, the smaller the scale size.

[0061] After obtaining the first feature map at the N-level scale of the original image output by the multi-scale feature extraction module, the first feature map at the N-level scale can be input into the target fusion module of the semantic segmentation model. The target fusion module performs at least one fusion process on the first feature map at the N-level scale to obtain the target fusion feature map at the N-level scale output by the target fusion module.

[0062] The method for obtaining the target fusion feature map at the N-level scale after the target fusion module performs at least one fusion process on the first feature map at the N-level scale can be as follows: first, the first fusion module performs a first fusion process on the first feature map at the N-level scale to obtain the first fusion feature map at the N-level scale output by the first fusion module; then, the second fusion module performs a second fusion process on the first fusion feature map at the N-level scale to obtain the second fusion feature map at the N-level scale output by the second fusion module. The target fusion module includes a first fusion module and a second fusion module, and the second fusion feature map at the N-level scale is the target fusion feature map at the N-level scale.

[0063] The method for the first fusion module to perform a first fusion process on the first feature map at the N-level scale to obtain the first fused feature map at the N-level scale output by the first fusion module can be as follows: For the first fused feature map at the (n-1)-level scale, firstly, the first feature map at the n-level scale is input into the upsampling module of the semantic segmentation model, and the upsampling module performs upsampling processing to obtain the upsampled feature map at the n-level scale; where 2≤n≤N; then, the first fusion module performs a first fusion process on the upsampled feature map at the n-level scale and the first feature map at the (n-1)-level scale to obtain the first fused feature map at the (n-1)-level scale output by the first fusion module; for the first fused feature map at the n-level scale, the first feature map at the n-level scale is used as the first fused feature map at the n-level scale.

[0064] Please continue reading. Figure 2 ,Depend on Figure 2It can be seen that for the first fused feature map at the first level scale, the first feature map at the second level scale (1 / 16 of the first feature map) can be input into the upsampling module of the semantic segmentation model. The upsampling module performs upsampling processing to obtain the second-level scale upsampled feature map (the 1 / 8 upsampled feature map obtained after upsampling the 1 / 16 first feature map). Then, the first fusion module performs the first fusion processing on the second-level scale upsampled feature map and the first feature map at the first level scale to obtain the first fused feature map at the first level scale output by the first fusion module. It can be understood that the method for obtaining the first fused feature map at the second, third, and fourth levels is similar to the method for calculating the first fused feature map at the first level scale, and will not be elaborated here. For the first fused feature map at the fifth level scale, the first feature map at the fifth level scale can be used as the first fused feature map at the fifth level scale.

[0065] The method for the second fusion module to perform a second fusion process on the first fusion feature map at the N-level scale to obtain the second fusion feature map at the N-level scale output by the second fusion module can be as follows: For the second fusion feature map at the 1-level scale, the first fusion feature map at the 1-level scale is used as the second fusion feature map at the 1-level scale; for the second fusion feature map at the n-level scale, the first feature map at the (n-1)-level scale is first input into the downsampling module of the semantic segmentation model, and the downsampling module performs downsampling processing to obtain the downsampled feature map at the (n-1)-level scale; where 2≤n≤N; then the second fusion module performs a second fusion process on the downsampled feature map at the (n-1)-level scale and the first fusion feature map at the n-level scale to obtain the second fusion feature map at the n-level scale output by the second fusion module.

[0066] Please continue reading. Figure 2 ,Depend on Figure 2 It can be seen that for the second fusion feature map at the first level scale, the first fusion feature map at the first level scale can be used as the second fusion feature map at the first level scale. For the second fusion feature map at the second level scale, the first feature map at the first level scale is first input into the downsampling module of the semantic segmentation model, and the downsampling module performs downsampling processing to obtain the downsampled feature map at the first level scale. Then, the second fusion module performs a second fusion processing on the downsampled feature map at the first level scale and the first fusion feature map at the second level scale to obtain the second fusion feature map at the second level scale output by the second fusion module. The method for obtaining the second fusion feature map at the third, fourth, and fifth levels is similar to the method for obtaining the second fusion feature map at the second level scale, and will not be described in detail here.

[0067] It is worth mentioning that performing two fusion processes improves the comprehensiveness and accuracy of image feature extraction and improves segmentation accuracy, that is, it improves the intersection-union ratio of the semantic segmentation image output by the model and the semantic segmentation image of the label.

[0068] After obtaining the N-level scale target fusion feature map output by the target fusion module, a semantic segmentation image corresponding to the original image can be generated based on the N-level scale target fusion feature map.

[0069] One method for generating the semantic segmentation image corresponding to the original image based on the target fusion feature map at an N-level scale can be to input the target fusion feature map at an N-level scale into the global convolutional module of the semantic segmentation model, thereby obtaining the second feature map at an N-level scale output by the global convolutional module. (See also...) Figure 3 , Figure 3 This is a schematic diagram of the structure of a global convolution module disclosed in an embodiment of this application. Figure 3 The 14 in the equation represents the number of semantic categories. The global convolutional module has two branches: 1*k + k*1. It's important to understand that replacing k*k with this method, without following a non-linear activation function, reduces computation while maintaining a certain receptive field. This improves detection speed while ensuring accuracy, thus enhancing timeliness. Furthermore, replacing fully connected layers and global pooling with the global convolutional module reduces the possibility of losing localization information and improves localization performance.

[0070] After obtaining the second feature map at scale N output from the global convolutional module, this second feature map can be input into the edge refinement module of the semantic segmentation model to obtain the third feature map at scale N output from the edge refinement module. Then, the semantic segmentation image corresponding to the original image can be generated based on the third feature map. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of the structure of an edge refinement module disclosed in an embodiment of this application. Figure 4 The structure is a residual connection. The top w*h*14 is a coarse feature map, and the residual connections on the sides can improve the boundary of the recognized object, achieving the effect of edge refinement, thereby improving the IOU of the semantic segmentation image.

[0071] It's worth noting that before inputting the original image into the multi-scale feature extraction module of the semantic segmentation model, the semantic segmentation model can be trained. The training method involves first obtaining original image samples, each labeled with a semantic segmentation image; then inputting these original image samples into the semantic segmentation model to obtain the predicted semantic segmentation image corresponding to the original image sample output by the model. Finally, the loss between the predicted and labeled semantic segmentation images is calculated using a regression loss function. When the loss meets the convergence condition, the trained semantic segmentation model is obtained. Specifically, this can be achieved using the Adam optimizer with a learning rate of 0.001.

[0072] It is worth mentioning that, compared to the existing Local Pyramid Occupancy Network (PON), the improved pyramid feature extraction network in this embodiment provides more comprehensive semantic extraction and produces higher accuracy results. Under conditions where the computer uses a GPU (NVIDIA RTX 3090), a single detection can be completed in just 0.2 seconds. Compared to the monocular semantic occupancy mesh mapping (VED) method using a convolutional V-fractal encoder-decoder network—which uses RGB video as input and a single encoder-decoder network model to predict semantic segmentation images—this embodiment improves the comprehensiveness of feature extraction and the accuracy of semantic segmentation, enhancing the semantic segmentation effect for dynamic objects, making it applicable to real-world driving scenarios. Compared to the cross-view semantic segmentation (VPN) method, which uses a view relationship module to establish surrounding objects for semantic segmentation, this embodiment improves the integrity of spatial information, ensuring the accuracy of semantic segmentation for small targets as well as large target points, thereby improving the overall accuracy of semantic segmentation and increasing IOU.

[0073] It is understood that, in addition to the methods described above, such as the method of performing at least one fusion process on the first feature map at the N-level scale by the target fusion module, the method of performing a first fusion process on the first feature map at the N-level scale by the first fusion module, the method of performing a second fusion process on the first fused feature map at the N-level scale by the second fusion module, and the method of generating a semantic segmentation image corresponding to the original image based on the target fusion feature map at the N-level scale, other reasonable methods may also be used, and no specific limitations are made here.

[0074] In this embodiment, after obtaining the first feature map at an N-level scale of the original image, the first feature map at an N-level scale can be input into the target fusion module of the semantic segmentation model. The target fusion module performs at least one fusion process on the first feature map at the N-level scale, improving the comprehensiveness and accuracy of image feature extraction and increasing segmentation precision, i.e., improving the intersection-union ratio (IOU) between the semantic segmentation image output by the model and the semantic segmentation image with labels. Secondly, the target fused feature map at the N-level scale can be input into the global convolution module of the semantic segmentation model for feature extraction, obtaining the second feature map at an N-level scale output by the global convolution module. This improves the comprehensiveness and accuracy of image feature extraction, solves the problem of imperfect use of feature layers in the model, and thus improves segmentation precision and IOU. Furthermore, the second feature map at an N-level scale output by the global convolution module can be input into the edge refinement module, obtaining the third feature map at an N-level scale output by the edge refinement module, improving the edge refinement degree and thus improving segmentation precision. Finally, by performing a series of feature extractions through a multi-scale feature extraction module, a target fusion module, a global convolution module, and an edge refinement module, the completeness of the semantic segmentation model can be improved, and the speed of semantic segmentation can be increased while extracting more semantic features.

[0075] Please refer to the following: Figure 5 One embodiment of the image processing device 500 in this application includes:

[0076] Central processing unit 501, memory 505, input / output interface 504, wired or wireless network interface 503, and power supply 502;

[0077] Memory 505 is either a short-term storage memory or a persistent storage memory;

[0078] The central processing unit 501 is configured to communicate with the memory 505 and execute instructions stored in the memory 505 to perform the aforementioned operations. Figure 1 The method in the illustrated embodiment.

[0079] This application also provides a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the aforementioned actions. Figure 1 The method in the illustrated embodiment.

[0080] This application also provides a computer program product containing instructions, which, when run on a computer, causes the computer to perform the aforementioned... Figure 1 The method in the illustrated embodiment.

[0081] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0086] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A semantic segmentation method, characterized in that, include: After obtaining the original image, the original image is input into the multi-scale feature extraction module of the semantic segmentation model to obtain the first feature map of the original image at N-level scale output by the multi-scale feature extraction module; where N is an integer greater than or equal to 2; and the semantic segmentation model is a pre-trained model. The first feature map at the N-level scale is input into the target fusion module of the semantic segmentation model. The target fusion module performs at least one fusion process on the first feature map at the N-level scale to obtain the target fusion feature map at the N-level scale output by the target fusion module. Generate a semantic segmentation image corresponding to the original image based on the target fusion feature map at the N-level scale; The target fusion module includes a first fusion module and a second fusion module; The process of performing at least one fusion process on the first feature map at the N-level scale by the target fusion module to obtain the target fusion feature map at the N-level scale output by the target fusion module includes: The first fusion module performs a first fusion process on the first feature map at the N-level scale to obtain the first fused feature map at the N-level scale output by the first fusion module. The second fusion module performs a second fusion process on the first fusion feature map at the N-level scale to obtain the second fusion feature map at the N-level scale output by the second fusion module; wherein, the second fusion feature map at the N-level scale is the target fusion feature map at the N-level scale; The first fusion process, in which the first fusion module performs a first fusion process on the first feature map at the N-level scale to obtain the first fused feature map at the N-level scale output by the first fusion module, includes: For the first fused feature map at the (n-1)th scale, the first feature map at the nth scale is input into the upsampling module of the semantic segmentation model, and the upsampling module performs upsampling processing to obtain the upsampled feature map at the nth scale; where 2≤n≤N; The first fusion module performs a first fusion process on the upsampled feature map at the nth level scale and the first feature map at the (n-1)th level scale to obtain the first fused feature map at the (n-1)th level scale output by the first fusion module. For the first fused feature map at the nth level scale, the first feature map at the nth level scale is used as the first fused feature map at the nth level scale. The second fusion module performs a second fusion process on the first fusion feature map at the N-level scale to obtain the second fusion feature map at the N-level scale output by the second fusion module, including: For the second fused feature map at the first level scale, the first fused feature map at the first level scale is used as the second fused feature map at the first level scale; For the second fused feature map at the nth level scale, the first feature map at the (n-1)th level scale is input into the downsampling module of the semantic segmentation model, and the downsampling module performs downsampling processing to obtain the downsampled feature map at the (n-1)th level scale; where 2≤n≤N; The second fusion module performs a second fusion process on the downsampled feature map at the (n-1)th scale and the first fusion feature map at the nth scale to obtain the second fusion feature map at the nth scale output by the second fusion module.

2. The method according to claim 1, characterized in that, Before inputting the original image into the multi-scale feature extraction module of the semantic segmentation model, the method further includes: The original image is preprocessed to obtain a preprocessed original image; The multi-scale feature extraction module that inputs the original image into the semantic segmentation model includes: The preprocessed original image is input into the multi-scale feature extraction module.

3. The method according to claim 1, characterized in that, The step of generating the semantic segmentation image corresponding to the original image based on the target fusion feature map at the N-level scale includes: The target fusion feature map at the N-level scale is input into the global convolutional module of the semantic segmentation model to obtain the second feature map at the N-level scale output by the global convolutional module. The semantic segmentation image corresponding to the original image is generated based on the second feature map at the N-level scale.

4. The method according to claim 3, characterized in that, The step of generating the semantic segmentation image corresponding to the original image based on the second feature map at the N-level scale includes: The second feature map at the N-level scale is input into the edge refinement module of the semantic segmentation model to obtain the third feature map at the N-level scale output by the edge refinement module. The semantic segmentation image corresponding to the original image is generated based on the third feature map.

5. The method according to claim 1, characterized in that, Before inputting the original image into the multi-scale feature extraction module of the semantic segmentation model, the method further includes: Obtain original image samples; where each original image sample is labeled with a semantic segmentation image; The original image sample is input into the semantic segmentation model to obtain the predicted semantic segmentation image corresponding to the original image sample output by the semantic segmentation model; The loss between the predicted semantic segmentation image and the labeled semantic segmentation image is calculated based on the regression loss function. When the loss satisfies the convergence condition, the trained semantic segmentation model is obtained.

6. An image processing device, characterized in that, include: Central processing unit, memory, input / output interfaces, wired or wireless network interfaces, and power supply; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 5.

Images