Image semantic segmentation method and device, electronic equipment and nonvolatile storage medium

By using the branch extraction feature map and hollow convolution fusion method of repeated execution of convolutional networks in small devices, the problem of high device configuration requirements of deep convolutional neural networks is solved, and the efficiency, accuracy and real-time image semantic segmentation are achieved. .

CN119068183APending Publication Date: 2024-12-03THE GBA NAT INST FOR NANOTECHNOLOGY INNOVATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310645614.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, the image semantic segmentation method relies on deep convolutional neural networks, resulting in high requirements for device configuration and cannot be effectively implemented in small mobile devices such as mobile phones and drones.

Method used

The method of repeatedly performing the branch extraction feature map of the convolution network is adopted, combining hollow convolution and upsampling fusion to reduce the calculation amount and realize semantic segmentation, which is suitable for small devices.

Benefits of technology

The efficient operation of image semantic segmentation is realized in small devices, reducing hardware requirements, and ensuring the accuracy and real-time nature of semantic segmentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068183A_ABST
    Figure CN119068183A_ABST
Patent Text Reader

Abstract

The invention discloses an image semantic segmentation method and device, electronic equipment and a nonvolatile storage medium. The method comprises the following steps: repeatedly executing a step of extracting a feature map of a first to-be-processed image through a first branch and a second branch in a first preset convolution network for a first preset number of times to obtain a feature map set, the first branch comprising a first convolution kernel and a second convolution kernel which are connected in series, and the second branch comprising a plurality of cavity convolution kernels; repeatedly executing the step of up-sampling a second to-be-processed image for a second preset number of times and fusing the second to-be-processed image with the previous feature map, and extracting a feature map of the fused feature map through a third branch and a fourth branch in a second preset convolutional network to obtain a target feature map; and determining a semantic segmentation result of the original image according to the target feature map. According to the method and the device, the technical problem that the image semantic segmentation method cannot be realized in small mobile equipment due to the fact that the configuration requirement on hardware equipment is too high when image semantic segmentation is carried out in related technologies is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to an image semantic segmentation method, device, electronic device and non-volatile storage medium. Background Art

[0002] At present, the image semantic segmentation methods in related technologies are usually implemented based on deep convolutional neural network models. Although image semantic segmentation can be achieved through the neural network structure, it has high requirements on the configuration of the device. For small and limited-configuration devices such as mobile phones and drones in actual application scenarios, due to insufficient computing power, it is impossible to directly use deep convolutional neural network models in common devices to complete image semantic segmentation.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an image semantic segmentation method, device, electronic device and non-volatile storage medium to at least solve the technical problem that the image semantic segmentation method cannot be implemented in a small mobile device due to the excessively high configuration requirements for hardware devices when performing image semantic segmentation in the related technology.

[0005] According to one aspect of an embodiment of the present application, a method for image semantic segmentation is provided, comprising: repeatedly executing the step of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed whose feature map is extracted for the first time through the first preset convolutional network is the original image, and the first feature map to be processed which is not the first time to extract the feature map through the first preset convolutional network is an image obtained by downsampling the feature map extracted by the first preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted each time through the first preset convolutional network; repeatedly executing the second preset Suppose a number of steps of upsampling the second image to be processed and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting the feature map of the fused feature map through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network after the last upsampling, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels; determine the semantic segmentation result of the original image according to the target feature map.

[0006] Optionally, the first preset convolution network also includes a fifth convolution kernel, and the step of extracting a feature map of the first image to be processed through the first branch and the second branch in the first preset convolution network includes: performing dimensionality reduction convolution processing on the first image to be processed through the fifth convolution kernel to obtain a first intermediate feature map; continuously convolving the first intermediate feature map through the first convolution kernel and the second convolution kernel in the first branch to obtain a first convolution result; performing multiple dilated convolutions on the first intermediate feature map through multiple dilated convolution kernels in the second branch to obtain a second convolution result; and fusing the first image to be processed, the first convolution result, and the second convolution result to obtain a feature map corresponding to the image obtained after downsampling.

[0007] Optionally, the second branch includes a first dilated convolution kernel, a second dilated convolution kernel and a third dilated convolution kernel; multiple dilated convolutions are performed on the image obtained after downsampling through multiple dilated convolution kernels in the second branch, and the step of obtaining a second convolution result includes: using the first dilated convolution kernel to perform a first dilated convolution processing on the first intermediate feature map to obtain a first dilated convolution result; using the second dilated convolution kernel to perform convolution processing on the first intermediate feature map, and merging the first dilated convolution result to obtain a second dilated convolution result; using the third dilated convolution kernel to perform convolution processing on the first intermediate feature map, and merging the second convolution result to obtain a third dilated convolution result, wherein the second convolution result includes the first dilated convolution result, the second dilated convolution result and the third dilated convolution result.

[0008] Optionally, the step of fusing the first image to be processed, the first convolution result and the second convolution result to obtain a feature map corresponding to the image obtained after downsampling includes: performing channel stitching processing on the first convolution result and the second convolution result to obtain a second intermediate feature map; determining the dependency relationship between each channel in the second intermediate feature map, and adjusting the feature response value of each channel according to the dependency relationship; fusing the second intermediate feature map after adjusting the feature response value and the first image to be processed to obtain a feature map corresponding to the first image to be processed.

[0009] Optionally, upsampling the second image to be processed and fusing it with the above feature map of the second image to be processed includes: determining a region of interest in the above feature map through a spatial attention network; and adjusting the fusion weights of each channel in the above feature map according to the region of interest in the above feature map.

[0010] Optionally, the second preset convolution network also includes a sixth convolution kernel, and the step of extracting a feature map of the fused feature map obtained by fusing the third branch and the fourth branch in the second preset convolution network includes: performing dimensionality reduction convolution processing on the fused feature map through the sixth convolution kernel to obtain a third intermediate feature map; continuously convolving the third intermediate feature map through the second convolution kernel and the third convolution kernel in the third branch to obtain a third convolution result; performing multiple dilated convolutions on the third intermediate feature map through multiple dilated convolution kernels in the fourth branch to obtain a fourth convolution result; merging the fused feature map, the third convolution result and the fourth convolution result to obtain a feature map corresponding to the fused feature map.

[0011] Optionally, the fourth branch includes a fourth dilated convolution kernel, a fifth dilated convolution kernel and a sixth dilated convolution kernel; multiple dilated convolutions are performed on the image obtained after downsampling through multiple dilated convolution kernels in the fourth branch, and the step of obtaining a fourth convolution result includes: using the fourth dilated convolution kernel to perform a first dilated convolution processing on the third intermediate feature map to obtain a fourth dilated convolution result; using the fifth dilated convolution kernel to perform convolution processing on the third intermediate feature map, and merging the fourth dilated convolution result to obtain a fifth dilated convolution result; using the sixth dilated convolution kernel to perform convolution processing on the third intermediate feature map, and merging the fifth dilated convolution result to obtain a sixth dilated convolution result, wherein the fourth convolution result includes the fourth dilated convolution result, the fifth dilated convolution result and the sixth dilated convolution result.

[0012] Optionally, the step of merging the fused feature map, the third convolution result and the fourth convolution result to obtain a feature map corresponding to the fused feature map includes: performing channel splicing processing on the third convolution result and the fourth convolution result to obtain a fourth intermediate feature map; determining the dependency between the channels in the fourth intermediate feature map, and adjusting the feature response values ​​of the channels based on the dependency; fusing the fourth intermediate feature map after adjusting the feature response values ​​and the fused feature map to obtain a feature map corresponding to the fused feature map.

[0013] Optionally, the feature map set includes a shallow feature map, a middle feature map, a deep feature map and a bottom feature map, wherein the resolution of the shallow feature map is equal to the resolution of the image to be processed, the resolution of the middle feature map is smaller than the resolution of the image to be processed, the resolution of the deep feature map is smaller than the resolution of the middle feature map, and the resolution of the bottom feature map is smaller than the resolution of the deep feature map.

[0014] According to another aspect of an embodiment of the present application, a method for training an image semantic segmentation model is also provided, comprising: obtaining a training data set, the training data set comprising a sample image and label information corresponding to the sample image; inputting the training data set into the image semantic segmentation model to be trained, and obtaining a predicted segmentation result output by the image semantic segmentation model to be trained, wherein the image semantic segmentation model to be trained comprises an encoding module and a decoding module, the encoding module comprises a plurality of first preset convolutional networks, the decoding module comprises a plurality of second preset convolutional networks, the first preset convolutional network comprises a first branch and a second branch, the first branch comprises a first convolutional kernel and a second convolutional kernel connected in series, the second branch comprises a plurality of hole convolutional kernels, the second preset convolutional network comprises a third branch and a fourth branch, the third branch comprises a third convolutional kernel and a fourth convolutional kernel connected in series, and the second branch comprises a plurality of hole convolutional kernels; after obtaining the predicted segmentation result each time, comparing the predicted segmentation result and the label information, and adjusting the model parameters of the image semantic segmentation model to be trained according to the comparison result, until the comparison result meets the preset requirements; determining that the image semantic segmentation model to be trained whose parameters have been adjusted for the last time is the trained image semantic segmentation model.

[0015] Optionally, the image semantic segmentation model to be trained also includes a spatial attention module, a feature fusion module, an upsampling module, and a downsampling module, wherein the spatial attention module is used to determine the region of interest in the image input to the spatial attention module, and adjust the fusion weights of the channels in the image input to the spatial attention module according to the region of interest; the feature fusion module is used to fuse the image input to the feature fusion module; the upsampling module is used to upsample the image input to the upsampling module; and the downsampling module is used to downsample the image input to the downsampling module.

[0016] Optionally, the first preset convolution network also includes a first dimensionality reduction convolution kernel and a first splicing module, and the second convolution network also includes a second dimensionality reduction convolution kernel and a second splicing module, wherein the first dimensionality reduction convolution kernel is used to perform dimensionality reduction convolution processing on the image input into the first convolution network; the second dimensionality reduction convolution kernel is used to perform dimensionality reduction convolution processing on the image input into the second convolution network; the first splicing module is used to fuse the result of the output of the first branch, the result of the output of each hole branch in the second branch, and the image input into the first preset convolution network to obtain a feature map output by the first preset convolution network; the second splicing module is used to fuse the result of the output of the third branch, the result of the output of each hole branch in the fourth branch, and the image input into the second preset convolution network to obtain a feature map output by the second preset convolution network.

[0017] Optionally, the first splicing module and the second splicing module also include a channel attention module, which is used to adjust the response characteristic values ​​of the channels in the image input to the channel attention module according to the dependency relationship between the channels in the image input to the channel attention module.

[0018] Optionally, the second branch includes a first hole branch, a second hole branch, and a third hole volume branch, wherein the first hole branch is used to perform hole convolution processing on the image output by the first dimensionality reduction convolution kernel and output it, the second hole branch is used to perform hole convolution processing on the image output by the first dimensionality reduction convolution kernel, and fuse the processed image with the output result of the first hole branch and output it, the third hole branch is used to perform hole convolution processing on the image output by the first dimensionality reduction convolution kernel, and fuse the processed image with the output result of the second hole branch and output it; the fourth branch includes a fourth hole branch, a fifth hole branch, and a sixth hole volume branch, wherein the fourth hole branch is used to perform hole convolution processing on the image output by the second dimensionality reduction convolution kernel and output it, the fifth hole branch is used to perform hole convolution processing on the image output by the second dimensionality reduction convolution kernel, and fuse the processed image with the output result of the fourth hole branch and output it, and the sixth hole branch is used to perform hole convolution processing on the image output by the second dimensionality reduction convolution kernel, and fuse the processed image with the output result of the fifth hole branch and output it.

[0019] According to another aspect of an embodiment of the present application, a method for image semantic segmentation is also provided, including: obtaining an original image; repeatedly executing the step of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed that extracts a feature map through the first preset convolutional network for the first time is the original image, and the first image to be processed that does not extract a feature map through the first preset convolutional network for the first time is an image obtained by downsampling the feature map extracted by the preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes multiple hole convolution kernels, and the feature map set includes feature maps extracted each time through the first preset convolutional network; repeatedly executing the step of extracting a feature map of a first image to be processed through a first preset convolutional network for a first preset number of times to obtain a feature map set. The invention further comprises the steps of upsampling the second image to be processed a preset number of times and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting the feature map of the fused feature map through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network during the last upsampling process, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes the first convolution kernel and the second convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels; the semantic segmentation result of the original image is determined according to the target feature map.

[0020] According to another aspect of an embodiment of the present application, an image semantic segmentation device is also provided, including: a first processing module, used to repeatedly execute the step of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed whose feature map is extracted for the first time through the first preset convolutional network is the original image, and the first feature map to be processed which is not the first time to extract the feature map through the first preset convolutional network is an image obtained by downsampling the feature map extracted by the first preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted each time through the first preset convolutional network; a second processing module, used to repeatedly The steps of upsampling the second image to be processed for a second preset number of times and fusing it with the above feature map of the second image to be processed are performed, and after each fusion, the feature map of the fused feature map is extracted by the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network after the last upsampling, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels; a third processing module is used to determine the semantic segmentation result of the original image according to the target feature map.

[0021] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute an image semantic segmentation method or an image semantic segmentation model training method.

[0022] According to another aspect of an embodiment of the present application, an electronic device is also provided, including: a memory and a processor, the processor being used to run a program stored in the memory, wherein the program executes an image semantic segmentation method and an image semantic segmentation model training method when the program is running.

[0023] In an embodiment of the present application, the step of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network is repeated for a first preset number of times to obtain a feature map set, wherein the first image to be processed that extracts a feature map through the first preset convolutional network for the first time is an original image, and the first feature map to be processed that is not the first time to extract a feature map through the first preset convolutional network is an image obtained by downsampling the feature map extracted by the first preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted by the first preset convolutional network each time; the step of upsampling the second image to be processed and fusing it with the above feature map of the second image to be processed is repeated for a second preset number of times, and after each fusion, the third branch and the fourth branch in the second preset convolutional network are used to extract a feature map of the first image to be processed. Take the feature map of the fused feature map to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolution network after the last upsampling, the target feature map is the feature map extracted by the second preset convolution network after the last upsampling, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels; according to the method of determining the semantic segmentation result of the original image based on the target feature map, by adopting the hole depth separable convolution method in the feature extraction and feature fusion steps, the purpose of reducing the computational amount of the convolution process is achieved, thereby solving the technical problem that the image semantic segmentation method cannot be implemented in a small mobile device due to the excessively high configuration requirements for the hardware device when performing image semantic segmentation in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 is a structural schematic diagram of a computer device according to an embodiment of the present application;

[0026] Figure 2 is a flowchart of an image semantic segmentation method according to an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of a convolution process of a depthwise separable convolution according to an embodiment of the present application;

[0028] Figure 4 is a flowchart of an image semantic segmentation process according to an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of an image semantic segmentation result according to an embodiment of the present application;

[0030] Figure 6 It is a flowchart of an image semantic segmentation model training method according to an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of the structure of a first preset convolutional network according to an embodiment of the present application;

[0032] Figure 8 It is a schematic diagram of a workflow of a first preset convolutional network and a second preset convolutional network according to an embodiment of the present application;

[0033] Fig. 9 is a schematic diagram of the principles of a channel attention module and a spatial attention module according to an embodiment of the present application;

[0034] Fig.10 is a schematic diagram of another image semantic segmentation result according to an embodiment of the present application;

[0035] Fig.11 is a flowchart of another image semantic segmentation method according to an embodiment of the present application;

[0036] Fig.12 It is a structural schematic diagram of an image semantic segmentation device according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0040] Semantic segmentation: Semantic segmentation is a typical computer vision problem that involves taking some raw data (such as a plane image, etc.) as input and converting the input raw data into a mask with highlighted regions of interest. In addition, semantic segmentation can group together parts of the image that belong to the same object.

[0041] In the related art, the semantic segmentation of images is usually achieved based on deep convolutional neural networks, and all of them include encoding and decoding architectures. The encoder can be understood as the process of downsampling the image and extracting the semantic information of the image, and the decoding can be understood as the process of upsampling the image features obtained by the encoder to obtain an output image with the same dimension as the input image. Although a certain degree of calculation accuracy can be obtained by using a deep convolutional neural network, this method has high requirements on the configuration of the equipment and is not suitable for use in small mobile devices with limited storage resources and weak computing power, such as mobile phones, drones, robots, etc., which are common in actual application scenarios. In order to achieve semantic segmentation tasks with accuracy that meets the requirements in conventional small mobile devices, a relevant solution is provided in the embodiments of the present application, which is described in detail below.

[0042] According to an embodiment of the present application, a method embodiment of an image semantic segmentation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing an image semantic segmentation method or an image semantic segmentation model training method is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0044] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0045] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image semantic segmentation method or the image semantic segmentation model training method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the image semantic segmentation method or the image semantic segmentation model training method of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0047] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0048] In the above operating environment, the present application embodiment provides an image semantic segmentation method, such as Figure 2 As shown, the method comprises the following steps:

[0049] Step S202, repeatedly performing the step of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed for which a feature map is extracted through the first preset convolutional network for the first time is an original image, and the first feature map to be processed for which a feature map is not extracted through the first preset convolutional network for the first time is an image obtained by downsampling the feature map extracted through the first preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted through the first preset convolutional network each time;

[0050] In the technical solution provided in step S202, the above-mentioned feature map set includes a shallow feature map, a middle feature map, a deep feature map and a bottom feature map, wherein the resolution of the shallow feature map is equal to the resolution of the image to be processed, the resolution of the middle feature map is less than the resolution of the image to be processed, the resolution of the deep feature map is less than the resolution of the middle feature map, and the resolution of the bottom feature map is less than the resolution of the deep feature map. The first preset number can be set by the staff according to actual needs, and the number of the first preset convolutional networks is determined according to the set first preset number.

[0051] It should be noted that the resolution of the shallow feature map is the same as the resolution of the image to be processed, the resolution of the middle feature map is 1 / 2 of the image to be processed, the resolution of the deep feature map is 1 / 4 of the image to be processed, and the resolution of the bottom feature map is 1 / 8 of the image to be processed. In addition, the above-mentioned first preset convolutional network and the second preset convolutional network can also be referred to as a hole residual module or a hole depth separable residual module. In addition, taking the shallow feature map as an example, the set of feature maps or feature maps described in this application refers to all channels contained in the shallow feature map, that is, all feature maps obtained after each convolution or other operation. In the embodiments of the present application, unless otherwise specified, the feature maps refer to all feature maps obtained after a convolution or other operation.

[0052] Specifically, obtaining a feature map set through the first preset convolutional network includes the following steps: taking the original image as the input image of the first preset convolutional network, processing the image to be processed through the first convolution branch and the second convolution branch to obtain a shallow feature map; downsampling the shallow feature map to obtain a first downsampling result, and using the first downsampling result as the input image of the first preset convolutional network, processing the first downsampling result through the first convolution branch and the second convolution branch to obtain a middle-layer feature map; downsampling the middle-layer feature map to obtain a second downsampling result, and using the second downsampling result as the input image of the first preset convolutional network, processing the second downsampling result through the first convolution branch and the second convolution branch to obtain a deep feature map; downsampling the deep feature map to obtain a third downsampling result, and using the third downsampling result as the input image of the first preset convolutional network, processing the third downsampling result through the first convolution branch and the second convolution branch to obtain an underlying feature map.

[0053] Specifically, the step of obtaining the feature map can be expressed by the following formula, where Fin represents the image to be processed, F1 represents the shallow feature map, F2 represents the middle feature map, F3 represents the deep feature map, F4 represents the bottom feature map, Down() represents downsampling, and AD represents the convolution processing of the input image using the first preset convolution network:

[0054] F1=AD(Fin)

[0055] F2=AD(Down(F1))

[0056] F3=AD(Down(F2))

[0057] F4=AD(Down(F3))

[0058] It should be noted that the output image of the target neural network has more channels than the input image, that is to say, the target neural network can perform channel upscaling operations as needed when extracting features of the input image. The input image can be any one of the image to be processed, the shallow feature map after downsampling, the middle feature map after downsampling, and the deep feature map after downsampling.

[0059] As an optional implementation, in order to ensure semantic segmentation accuracy while reducing the amount of computation, the first preset convolutional network also includes a fifth convolution kernel. The step of extracting a feature map of the first image to be processed through the first branch and the second branch in the first preset convolutional network includes: performing dimensionality reduction convolution processing on the first image to be processed through the fifth convolution kernel to obtain a first intermediate feature map; continuously convolving the first intermediate feature map through the first convolution kernel and the second convolution kernel in the first branch to obtain a first convolution result; performing multiple dilated convolutions on the first intermediate feature map through multiple dilated convolution kernels in the second branch to obtain a second convolution result; and fusing the first image to be processed, the first convolution result, and the second convolution result to obtain a feature map corresponding to the image obtained after downsampling.

[0060] Specifically, the fifth convolution kernel is a dimensionality reduction convolution kernel, for example, it can be a 1×1 convolution kernel. The fifth convolution kernel can be used to perform a convolution dimensionality reduction operation on the input image data, and obtain output image data with a channel number of 1 / 4 of the input data after the convolution dimensionality reduction operation. After that, the first convolution branch and the second convolution branch will respectively perform convolution processing on the image data obtained after the convolution operation, wherein the first convolution branch will use the first convolution kernel and the second convolution kernel connected in series to perform convolution in sequence, thereby extracting the local semantic information of the input image. The first convolution kernel can be a 3×1 convolution kernel, and the second convolution kernel can be a 1×3 convolution kernel. The second convolution branch includes three hole convolution kernels of the same specification, for example, three hole convolution kernels of 3×3 and a hole rate of d, and d can be set by itself. The second convolution branch can perform continuous hole convolution on the input image data, thereby extracting the generalized semantic information of the input image data. It should be noted that the use of dilated convolution can expand the receptive field during the convolution process and capture image context information at multiple scales.

[0061] In some embodiments of the present application, the second branch includes a first dilated convolution kernel, a second dilated convolution kernel and a third dilated convolution kernel; multiple dilated convolutions are performed on the image obtained after downsampling through multiple dilated convolution kernels in the second branch, and the step of obtaining a second convolution result includes: using the first dilated convolution kernel to perform a first dilated convolution processing on the first intermediate feature map to obtain a first dilated convolution result; using the second dilated convolution kernel to perform convolution processing on the first intermediate feature map, and merging the first dilated convolution result to obtain a second dilated convolution result; using the third dilated convolution kernel to perform convolution processing on the first intermediate feature map, and merging the second convolution result to obtain a third dilated convolution result, wherein the second convolution result includes the first dilated convolution result, the second dilated convolution result and the third dilated convolution result.

[0062] It can be seen that when performing dilated convolution, starting from the second dilated convolution, each convolution will fuse the result of the previous convolution, that is, the convolution result of the previous convolution, so richer image semantic information can be obtained in the convolution process.

[0063] As an optional implementation, the step of fusing the first image to be processed, the first convolution result and the second convolution result to obtain a feature map corresponding to the image obtained after downsampling includes: performing channel stitching processing on the first convolution result and the second convolution result to obtain a second intermediate feature map; determining the dependency between the channels in the second intermediate feature map, and adjusting the feature response values ​​of the channels according to the dependency; fusing the second intermediate feature map after adjusting the feature response values ​​and the first image to be processed to obtain a feature map corresponding to the first image to be processed.

[0064] Specifically, when fusing feature maps, the channel attention mechanism is used to determine the dependencies between channels in each feature map, and adjust the feature response value of each channel according to the dependencies. From the perspective of global information, valuable feature channels can be amplified (increased feature response values) and useless feature channels can be suppressed (reduced feature response values), thereby greatly improving network performance.

[0065] Since the specific implementation steps of the convolutional neural network for processing the image to be processed, the shallow feature map after downsampling, the middle feature map after downsampling, and the deep feature map after downsampling are the same, the following takes the image to be processed as an example to reflect the specific workflow of the convolutional neural network through the following formula:

[0066] Fbr1=C1x3(C3x1(C1x1(Fin)))

[0067] Fbr2=C3x3,d(C1x1(Fin))

[0068] Fbr3=C3x3,d((C1x1(Fin))+Fbr2

[0069] Fbr4=C3x3,d((C1x1(Fin))+Fbr3

[0070] Fout=CAM(Concat(Fbr1,Fbr2,Fbr3,Fbr4))+Fin

[0071] In the above formula, C1×1() means using the third convolution kernel to perform dimensionality reduction convolution on the data, C3×1() means using the first convolution kernel to convolve the data, C1×3() means using the second convolution kernel to convolve the data, C3×3,d() means using the hole convolution kernel to convolve the data, the “+” in the formula means image fusion, Concat() means channel splicing of the image, the number of channels of the image after splicing is the same as the sum of the number of channels of each image before splicing, CAM() means the channel attention mechanism, that is, determining the dependency between the channels in the image data to be processed, and adaptively adjusting the feature response value of each channel according to the dependency, so as to increase the weight of the region of interest. Fin is the input image, for example, it can be the first image to be processed, C1x1(Fin) is the first intermediate feature map, Fbr1 is the second intermediate feature map, Fbr2 is the third intermediate feature map, Fbr3 is the fourth intermediate feature map, Fbr4 is the fifth intermediate feature map, Concat(Fbr1, Fbr2, Fbr3, Fbr4) is the sixth intermediate feature map, and Fout is the output image.

[0072] It can be understood that the first preset convolutional network and the second preset convolutional network provided in the embodiment of the present application also include a BN layer and a ReLU layer. Although the BN layer and the ReLU layer are not represented in the above formula in the embodiment of the present application for the sake of simplicity and clarity of public expression, it does not mean that the data is not processed by the BN layer and the ReLU layer in the first preset convolutional network and the second preset convolutional network provided in the embodiment of the present application.

[0073] In order to avoid the grid effect that may occur during multiple dilated convolutions, in some embodiments of the present application, the HFF multi-feature fusion method can be used during multiple dilated convolutions to avoid the grid effect.

[0074] It should also be noted that when the first image to be processed is subjected to feature extraction through the first preset convolutional network, the first image to be processed may also be subjected to up-channel processing as needed. For example, when the image features in the original image are extracted through the first preset convolutional network, the original image may be subjected to up-channel processing first, and then the original image after the up-channel processing may be input into the first preset convolutional network.

[0075] Step S204, repeatedly performing the steps of upsampling the second image to be processed a second preset number of times and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting a feature map of the fused feature map obtained by fusion through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is a feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is a feature map extracted by the second preset convolutional network after the last upsampling, the target feature map is a feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the fourth branch includes a plurality of hole convolution kernels;

[0076] The second preset number may be determined according to the number of feature maps in the feature map set.

[0077] It should be noted that the second preset convolution network has the same structure as the first preset convolution network, but the parameters of the convolution kernels may be different. For example, the fourth dilated convolution kernel, the fifth dilated convolution kernel, and the sixth dilated convolution kernel may have the same specifications as the first dilated convolution kernel, the second dilated convolution kernel, and the third dilated convolution kernel, all of which are 3×3 and have a dilated convolution kernel with a dilated rate of d, and d can be set by yourself. As an optional implementation, the fourth dilated convolution kernel, the fifth dilated convolution kernel, and the sixth dilated convolution kernel may be different dilated convolution kernels, or they may be the same convolution kernels, that is, the first dilated convolution kernel, the second dilated convolution kernel, and the third dilated convolution kernel may be directly used as the fourth dilated convolution kernel, the fifth dilated convolution kernel, and the sixth dilated convolution kernel.

[0078] In the technical solution provided in step S204, in order to protect the details of key areas and suppress useless areas in the feature fusion stage, a spatial attention network can be used to determine the fusion weights of the channels in each feature map, wherein the fusion weight can reflect the importance of the channel, and the larger the fusion weight, the more features in the channel are retained in the fusion stage. Specifically, upsampling the second image to be processed and fusing it with the above feature map of the second image to be processed includes: determining the region of interest in the above feature map through the spatial attention network; adjusting the fusion weights of each channel in the above feature map according to the region of interest in the above feature map.

[0079] As an optional implementation, the second preset convolution network also includes a sixth convolution kernel, and the step of extracting a feature map of the fused feature map obtained by fusing the third branch and the fourth branch in the second preset convolution network includes: performing dimensionality reduction convolution processing on the fused feature map through the sixth convolution kernel to obtain a third intermediate feature map; continuously convolving the third intermediate feature map through the second convolution kernel and the third convolution kernel in the third branch to obtain a third convolution result; performing multiple dilated convolutions on the third intermediate feature map through multiple dilated convolution kernels in the fourth branch to obtain a fourth convolution result; merging the fused feature map, the third convolution result and the fourth convolution result to obtain a feature map corresponding to the fused feature map.

[0080] In some embodiments of the present application, the fourth branch includes a fourth dilated convolution kernel, a fifth dilated convolution kernel and a sixth dilated convolution kernel; multiple dilated convolutions are performed on the image obtained after downsampling through multiple dilated convolution kernels in the fourth branch, and the step of obtaining a fourth convolution result includes: using the fourth dilated convolution kernel to perform a first dilated convolution processing on the third intermediate feature map to obtain a fourth dilated convolution result; using the fifth dilated convolution kernel to perform convolution processing on the third intermediate feature map, and merging the fourth dilated convolution result to obtain a fifth dilated convolution result; using the sixth dilated convolution kernel to perform convolution processing on the third intermediate feature map, and merging the fifth dilated convolution result to obtain a sixth dilated convolution result, wherein the fourth convolution result includes the fourth dilated convolution result, the fifth dilated convolution result and the sixth dilated convolution result.

[0081] As an optional implementation manner, the step of merging the fused feature map, the third convolution result and the fourth convolution result to obtain a feature map corresponding to the fused feature map includes: performing channel splicing processing on the third convolution result and the fourth convolution result to obtain a fourth intermediate feature map; determining the dependency between the channels in the fourth intermediate feature map, and adjusting the feature response values ​​of the channels based on the dependency; fusing the fourth intermediate feature map after adjusting the feature response values ​​and the fused feature map to obtain a feature map corresponding to the fused feature map.

[0082] In addition, the fusion weights of the channels in each feature map are adjusted through the spatial attention module, the target feature map is obtained through the second preset convolutional network, and the process of classifying and convolving the final target fusion image can be expressed by the following formula:

[0083] F 1fusion =SAM(F1)

[0084] F 2fusion =SAM(F2)

[0085] F 3fusion =SAM(F3)

[0086] F 1 / 4=AD(Up(F4)+F 3fusion )

[0087] F 1 / 2 =AD(Up(F1 / 4)+F 2fusion )

[0088] F out =C 3x3 (AD(Up(F 1 / 2 )+F 1fusion ))

[0089] In the above formula, SAM() represents the use of spatial attention network to process the data input to the network, F1 represents the shallow feature map, F2 represents the middle feature map, F3 represents the deep feature map, F4 represents the bottom feature map, and F 1fusion Represents the shallow feature map after adjusting the fusion weight of the channel, F 2fusion Represents the middle-level feature map after adjusting the fusion weight of the channel, F 3fusion Represents the shallow feature map after adjusting the fusion weight of the channel, Up(F4)+F 3fusion F represents the first fused feature map obtained by fusing the upsampled underlying feature map and the weighted deep feature map. 1 / 4 It represents the second fused feature map obtained after feature extraction of the first fused feature map, Up(F1 / 4)+F 2fusion represents the third fused feature map obtained by upsampling the second fused feature map and fusing it with the weighted middle feature map, F 1 / 2 It represents the fourth fusion feature map AD(Up(F 1 / 2 )+F 1fusion represents the target feature map obtained by fusion of the fourth fusion feature map after upsampling and the shallow feature map after weight adjustment, F out It represents the semantic segmentation result of the original image obtained after the target feature map is classified and convolved. UP() represents the upsampling operation, "+" represents image fusion, AD() represents the feature extraction processing of the input data using the second preset convolutional network, and C 3x3 () indicates that a 3×3 classification convolution kernel is used to perform classification convolution on the input image data. When upsampling, bilinear interpolation can be used.

[0090] It should be noted that the amount of calculation can be effectively reduced by performing separable convolution on the image to be processed through the first branch and the second branch in the first preset convolution network, or through the third branch and the fourth branch in the second preset convolution network. Specifically, assuming that an ordinary convolution network is used to convolve the image, the amount of calculation required to complete a convolution is A=Dk*Dk*M*N*Df*Df, and the amount of calculation required for separable convolution to complete a calculation is: B=Dk*Dk*M*Df*Df+M*N*Df*Df. Where Dk is the length and width of the convolution kernel, M is the number of channels of the image, and N is the number of convolution kernels and the number of channels of the output image. Therefore, α=B / A=1 / N+1 / (Dk*Dk), since Dk is usually selected as 3, the final α≈1 / 9, α is the ratio of the amount of calculation to complete the same operation using depth separable convolution and conventional convolution. It can be seen that the use of the first preset convolutional network and the second preset convolutional network provided in the present application can greatly reduce the amount of computation in the convolution process, thereby reducing the configuration requirements for executing the image semantic segmentation method. The process of using separable convolution to convolve the image data to be processed is as follows: Figure 3 As shown. Figure 3 It can be seen that in the convolution process of depthwise separable convolution, after processing the input image through multiple convolution kernels, the number of channels of the final output image will be more than the number of channels of the input image.

[0091] Step S206, determining the semantic segmentation result of the original image according to the target feature map.

[0092] As an optional implementation, the process of determining the semantic segmentation result of the original image according to the target feature map is as shown in the above formula “F out =C 3x3 (AD(Up(F 1 / 2 )+F 1fusion ))” and will not be described in detail here.

[0093] Through the above steps, the accuracy of semantic segmentation is guaranteed, the memory and computing conditions of the required hardware are reduced, and the real-time performance of the image semantic segmentation reasoning process is guaranteed. In addition, the present application provides a new feature extraction module, namely the hole depth separable convolution residual module, which can simultaneously ensure the acquisition of a larger receptive field and the protection of local information of features, and the amount of calculation is small. At the same time, the present application uses a U-shaped structure as a framework, which can realize the interaction of image context and restore the image segmentation details at multiple levels. Since the hardware requirements of the semantic segmentation method provided by this application are relatively low, it can be applied to mobile terminals such as drones, mobile phones, and robots, and can interact with people and objects around it in real time. It can also be applied to face recognition, medical treatment, automatic driving and other fields.

[0094] In addition, it can be seen from the above content that the image semantic segmentation method provided by this application will perform three downsampling and upsampling, and use skip connection in the same stage (image processing stage), instead of directly performing supervision and loss backpropagation on high-level semantic features. This ensures that the feature map finally recovered incorporates more low-level features, and also enables the fusion of features at different scales, so that multi-scale prediction and Deep Supervision can be performed. The three upsamplings also make the segmentation map restore edge information more refined. Specifically, using Figure 2 The image semantic segmentation method shown in the figure is used to perform semantic segmentation on the processed image. Figure 4 As shown, Figure 4 The spatial residual module in is the first preset convolutional network and the second preset convolutional network mentioned above, where: Figure 4 The spatial residual module in the down-sampling process is the first preset convolutional network, and the spatial residual module in the up-sampling process is the second preset convolutional network. The structures of the first preset convolutional network and the second preset convolutional network are the same, and only the parameters of the internal convolution kernel can be different or the same. Figure 4 It can be seen from the figure that in the downsampling stage, each time the feature map of the image to be processed is obtained through the spatial residual module, the fusion weights of the channels in the feature map are adjusted through the spatial attention module, and sent to the corresponding spatial residual module in the upsampling stage to realize feature fusion. Figure 4 It can be seen that in the downsampling stage, each time the spatial residual module is used to extract features, the number of channels of the image will increase, and in the upsampling stage, each time the spatial residual module is used to fuse features, the number of channels of the image will decrease until a semantic segmentation result image with the same number of channels as the image to be processed is obtained.

[0095] Specifically, the image semantic segmentation result obtained by the image semantic segmentation method provided in the embodiment of the present application is as follows: Figure 5 As shown, Figure 5 The left side is the input image. Figure 5 The right side of is the semantic segmentation result.

[0096] In the above operating environment, the present application embodiment provides an image semantic segmentation model training method, such as Figure 6 As shown, the method comprises the following steps:

[0097] Step S602, obtaining a training data set, wherein the training data set includes sample images and label information corresponding to the sample images;

[0098] In the technical solution provided in step S602, after obtaining the training data set, the training images and corresponding label information in the training data set need to be preprocessed to ensure that the training effect can be guaranteed even when the number of samples is limited. The preprocessing method may include flipping, rotation, translation, scaling / scaling, etc. Preprocessing the label information refers to adaptively adjusting the label information after the training image preprocessing.

[0099] Step S604, inputting the training data set into the image semantic segmentation model to be trained, and obtaining the predicted segmentation result output by the image semantic segmentation model to be trained, wherein the image semantic segmentation model to be trained includes an encoding module and a decoding module, the encoding module includes multiple first preset convolutional networks, the decoding module includes multiple second preset convolutional networks, the first preset convolutional network includes a first branch and a second branch, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes multiple hole convolution kernels, the second preset convolutional network includes a third branch and a fourth branch, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the second branch includes multiple hole convolution kernels;

[0100] Step S606, after obtaining the predicted segmentation result each time, compare the predicted segmentation result with the label information, and adjust the model parameters of the image semantic segmentation model to be trained according to the comparison result, until the comparison result meets the preset requirements;

[0101] Specifically, a cross entropy loss function can be calculated according to the comparison result as the objective function, and the model parameters of the image semantic segmentation model to be trained can be adjusted according to the objective function.

[0102] Step S608: determine that the image semantic segmentation model to be trained whose parameters are adjusted for the last time is the trained image semantic segmentation model.

[0103] In some implementations of the present application, the image semantic segmentation model to be trained also includes a spatial attention module, a feature fusion module, an upsampling module, and a downsampling module, wherein the spatial attention module is used to determine a region of interest in an image input to the spatial attention module, and adjust the fusion weights of channels in the image input to the spatial attention module according to the region of interest; the feature fusion module is used to fuse the image input to the feature fusion module; the upsampling module is used to upsample the image input to the upsampling module; and the downsampling module is used to downsample the image input to the downsampling module.

[0104] As an optional implementation, Figure 7 is a schematic diagram of the structure of a first preset convolutional network provided according to an embodiment of the present application, such as Figure 7As shown, the first preset convolution network also includes a first dimensionality reduction convolution kernel and a first splicing module, and the second convolution network also includes a second dimensionality reduction convolution kernel and a second splicing module, wherein the first dimensionality reduction convolution kernel is used to perform dimensionality reduction convolution processing on the image input into the first convolution network; the second dimensionality reduction convolution kernel is used to perform dimensionality reduction convolution processing on the image input into the second convolution network; the first splicing module is used to fuse the result of the output of the first branch, the result of the output of each hole branch in the second branch, and the image input into the first preset convolution network to obtain a feature map output by the first preset convolution network; the second splicing module is used to fuse the result of the output of the third branch, the result of the output of each hole branch in the fourth branch, and the image input into the second preset convolution network to obtain a feature map output by the second preset convolution network.

[0105] In order to retain more important channel features, such as Figure 7 As shown, the first splicing module and the second splicing module also include a channel attention module, which is used to adjust the response characteristic values ​​of the channels in the image input to the channel attention module according to the dependency relationship between the channels in the image input to the channel attention module. The structure of the second preset convolutional network is the same as that of the first preset convolutional network, so it will not be repeated here.

[0106] Specifically, the overall model structure of the image semantic segmentation model provided in the embodiments of the present application is as follows: Figure 4 As shown. It can be seen that the image semantic segmentation model provided in the embodiment of the present application is a U-shaped structure (such as a UNet architecture), and adopts a skip-layer connection method to merge the underlying position information with the deep semantic information through splicing, thereby retaining more dimension / position information. In addition, a spatial attention module can be used for connection during skip-layer connection, and the spatial attention module can obtain regions of interest in features at different levels.

[0107] In some embodiments of the present application, the second branch includes a first hole branch, a second hole branch, and a third hole volume branch, wherein the first hole branch is used to perform hole convolution processing on the image output by the first dimensionality reduction convolution kernel and output it, the second hole branch is used to perform hole convolution processing on the image output by the first dimensionality reduction convolution kernel, and fuse the processed image with the output result of the first hole branch and output it, the third hole branch is used to perform hole convolution processing on the image output by the first dimensionality reduction convolution kernel, and fuse the processed image with the output result of the second hole branch and output it; the fourth branch includes a fourth hole branch, a fifth hole branch, and a sixth hole volume branch, wherein the fourth hole branch is used to perform hole convolution processing on the image output by the second dimensionality reduction convolution kernel and output it, the fifth hole branch is used to perform hole convolution processing on the image output by the second dimensionality reduction convolution kernel, and fuse the processed image with the output result of the fourth hole branch and output it, and the sixth hole branch is used to perform hole convolution processing on the image output by the second dimensionality reduction convolution kernel, and fuse the processed image with the output result of the fifth hole branch and output it.

[0108] Specifically, the process of the first preset convolutional network convolving the image data to be processed is as follows: Figure 8 As shown. Figure 8 It can be seen that the first preset convolutional network and the second preset convolutional network both include four branches, one of which is a convolution branch with two convolution kernels connected in series, which is used to perform two consecutive convolution processes on the processed data, and the other three branches are hollow convolution branches, which are used to perform hollow convolution processes on the processed data, and the convolution results of other hollow convolution branches will be integrated during the processing process.

[0109] Fig. 9 A schematic diagram of the principle of the channel attention module in the first preset convolutional network and the second preset convolutional network and the spatial attention module in the model provided in the embodiment of the present application, wherein Fig. 9 The image above is the channel attention module. Fig. 9 The image below is the spatial attention module. Fig. 9 As can be seen from the figure, the channel attention module performs the maximum pooling operation (MaxPool) and the average pooling operation (AvgPool) on the input image, and inputs the pooled data into the shared multi-layer perceptron (Shared MLP) respectively, and fuses the output results of the shared multi-layer perceptron to obtain the response feature values ​​of each channel (Channel Attention, M c The spatial attention module also performs maximum pooling and average pooling operations on the input image, and then detects the target through multiple convolutional layers (conv layers) to obtain the spatial attention feature value (Spatial Attention, M s), which is the fusion weight of each channel described in this application.

[0110] In addition to being able to effectively reduce computing power requirements, the image semantic segmentation model provided by the present application is also superior to the image semantic segmentation methods in related technologies in terms of accuracy. Specifically, taking the damage detection task, that is, the segmentation task of metal welds and damages as an example, after obtaining the original data through X-ray detection, the three-dimensional reconstructed metal block image can be sliced, and the part to be segmented in the image can be labeled with labelme, and data preprocessing such as scaling and flipping can be performed. It is input into the image semantic segmentation model to be trained to obtain a trained image semantic segmentation model. The test data is then input into the trained image semantic segmentation model to obtain the semantic segmentation result. After testing, the image semantic segmentation model provided in the embodiment of the present application has a meanIOU of 87.2% for 100 test images. The input test data and the semantic segmentation results are shown in the figure. Fig.10 As shown, Fig.10 The left side is the test data, and the right side is the semantic segmentation result.

[0111] In terms of autonomous driving, when performing semantic segmentation tests on urban landscape datasets, the meanIOU of the image semantic segmentation model provided in the embodiment of the present application was 70.9%.

[0112] When testing on the Cityscapes dataset, the test results of the image semantic segmentation model provided by this application and the test results of other image semantic segmentation models in the related art are shown in the following table:

[0113]

[0114]

[0115] It can be seen that the image semantic segmentation model provided by the present application is still ahead of other algorithms in terms of accuracy without pre-training experimental processing methods, and does not affect the running speed. It can be said that the image semantic segmentation model provided by the present application achieves a balance between segmentation accuracy and parameter quantity.

[0116] In the above operating environment, the present application embodiment also provides an image semantic segmentation method, such as Fig.11 As shown, the method comprises the following steps:

[0117] Step S1102, obtaining an original image;

[0118] In the technical solution provided in step S1102, the original image obtained can be an image captured by an image acquisition module in a device running an image semantic segmentation model, such as an image captured by a camera of a small drone, mobile terminal device, etc., or it can be an original image obtained from other devices through wireless or wired communication lines.

[0119] Step S1104, repeatedly performing the step of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed for which a feature map is extracted through the first preset convolutional network for the first time is an original image, and the first image to be processed for which a feature map is not extracted through the first preset convolutional network for the first time is an image obtained by downsampling the feature map extracted through the preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted through the first preset convolutional network each time;

[0120] Step S1106, repeatedly performing the steps of upsampling the second image to be processed a second preset number of times and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting a feature map of the fused feature map obtained by fusion through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network in the last upsampling process, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes the first convolution kernel and the second convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels;

[0121] Step S1108, determining the semantic segmentation result of the original image according to the target feature map.

[0122] It should be noted that since the image semantic segmentation model provided in the embodiment of the present application has low configuration requirements and can ensure sufficient segmentation accuracy, Fig.11 The image semantic segmentation method shown in can be performed by small mobile devices with limited storage resources and weak computing power, such as mobile phones, drones, robots, etc., which are common in actual application scenarios. It can also be used to complete tasks such as face recognition, medical treatment, and autonomous driving.

[0123] The present application embodiment provides an image semantic segmentation device, Fig.12 is a schematic diagram of the structure of the device, such as Fig.12As shown, the device includes: a first processing module 120, which is used to repeatedly perform the steps of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed that extracts a feature map through the first preset convolutional network for the first time is an original image, and the first feature map to be processed that is not the first time to extract a feature map through the first preset convolutional network is an image obtained by downsampling the feature map extracted by the first preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted by each time through the first preset convolutional network; a second processing module 122, which is used to repeatedly perform the steps of extracting a feature map of a first image to be processed for a second preset number of times to obtain a feature map set. The steps of upsampling the second image to be processed and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting the feature map of the fused feature map through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network after the last upsampling, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels; a third processing module 124 is used to determine the semantic segmentation result of the original image based on the target feature map.

[0124] In some embodiments of the present application, the feature map set includes a shallow feature map, a middle feature map, a deep feature map and a bottom feature map, wherein the resolution of the shallow feature map is equal to the resolution of the image to be processed, the resolution of the middle feature map is smaller than the resolution of the image to be processed, the resolution of the deep feature map is smaller than the resolution of the middle feature map, and the resolution of the bottom feature map is smaller than the resolution of the deep feature map.

[0125] In some embodiments of the present application, the first preset convolution network also includes a fifth convolution kernel, and the step of the first processing module 120 extracting a feature map of the first image to be processed through the first branch and the second branch in the first preset convolution network includes: performing dimensionality reduction convolution processing on the first image to be processed through the fifth convolution kernel to obtain a first intermediate feature map; continuously convolving the first intermediate feature map through the first convolution kernel and the second convolution kernel in the first branch to obtain a first convolution result; performing multiple dilated convolutions on the first intermediate feature map through multiple dilated convolution kernels in the second branch to obtain a second convolution result; and fusing the first image to be processed, the first convolution result, and the second convolution result to obtain a feature map corresponding to the image obtained after downsampling.

[0126] In some embodiments of the present application, the second branch includes a first dilated convolution kernel, a second dilated convolution kernel and a third dilated convolution kernel; the first processing module 120 performs multiple dilated convolutions on the image obtained after downsampling through multiple dilated convolution kernels in the second branch, and the step of obtaining a second convolution result includes: using the first dilated convolution kernel to perform a first dilated convolution processing on the first intermediate feature map to obtain a first dilated convolution result; using the second dilated convolution kernel to perform convolution processing on the first intermediate feature map, and merging the first dilated convolution result to obtain a second dilated convolution result; using the third dilated convolution kernel to perform convolution processing on the first intermediate feature map, and merging the second convolution result to obtain a third dilated convolution result, wherein the second convolution result includes the first dilated convolution result, the second dilated convolution result and the third dilated convolution result.

[0127] In some embodiments of the present application, the first processing module 120 fuses the first image to be processed, the first convolution result and the second convolution result to obtain a feature map corresponding to the image obtained after downsampling, and the steps include: performing channel stitching processing on the first convolution result and the second convolution result to obtain a second intermediate feature map; determining the dependency between each channel in the second intermediate feature map, and adjusting the feature response value of each channel according to the dependency; fusing the second intermediate feature map after adjusting the feature response value and the first image to be processed to obtain a feature map corresponding to the first image to be processed.

[0128] In some embodiments of the present application, the second processing module 122 upsamples the second image to be processed and fuses it with the above feature map of the second image to be processed, including: determining the area of ​​interest in the above feature map through a spatial attention network; and adjusting the fusion weights of each channel in the above feature map based on the area of ​​interest in the above feature map.

[0129] In some embodiments of the present application, the second preset convolution network also includes a sixth convolution kernel, and the steps of extracting the feature map of the fused feature map obtained by the second processing module 122 through the third branch and the fourth branch in the second preset convolution network include: performing dimensionality reduction convolution processing on the fused feature map through the sixth convolution kernel to obtain a third intermediate feature map; continuously convolving the third intermediate feature map through the second convolution kernel and the third convolution kernel in the third branch to obtain a third convolution result; performing multiple dilated convolutions on the third intermediate feature map through multiple dilated convolution kernels in the fourth branch to obtain a fourth convolution result; merging the fused feature map, the third convolution result and the fourth convolution result to obtain a feature map corresponding to the fused feature map.

[0130] In some embodiments of the present application, the fourth branch includes a fourth dilated convolution kernel, a fifth dilated convolution kernel and a sixth dilated convolution kernel; the second processing module 122 performs multiple dilated convolutions on the image obtained after downsampling through multiple dilated convolution kernels in the fourth branch, and the step of obtaining a fourth convolution result includes: using the fourth dilated convolution kernel to perform a first dilated convolution processing on the third intermediate feature map to obtain a fourth dilated convolution result; using the fifth dilated convolution kernel to perform convolution processing on the third intermediate feature map, and merging the fourth dilated convolution result to obtain a fifth dilated convolution result; using the sixth dilated convolution kernel to perform convolution processing on the third intermediate feature map, and merging the fifth dilated convolution result to obtain a sixth dilated convolution result, wherein the fourth convolution result includes the fourth dilated convolution result, the fifth dilated convolution result and the sixth dilated convolution result.

[0131] In some embodiments of the present application, the second processing module 122 merges the fused feature map, the third convolution result and the fourth convolution result to obtain a feature map corresponding to the fused feature map, and the steps include: performing channel splicing processing on the third convolution result and the fourth convolution result to obtain a fourth intermediate feature map; determining the dependency between the channels in the fourth intermediate feature map, and adjusting the feature response values ​​of the channels based on the dependency; fusing the fourth intermediate feature map after adjusting the feature response values ​​and the fused feature map to obtain a feature map corresponding to the fused feature map.

[0132] It should be noted that the various modules in the above-mentioned image semantic segmentation device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0133] In addition, since the above-mentioned image semantic segmentation device can be used to perform Figure 2 The image semantic segmentation method shown in Figure 2 The relevant explanations and descriptions of the image semantic segmentation method shown in are also applicable to the above-mentioned image semantic segmentation device and will not be repeated here.

[0134] The embodiment of the present application provides a non-volatile storage medium, in which a program is stored, wherein when the program is running, a device where the non-volatile storage medium is located is controlled to execute Figure 2 The image semantic segmentation method shown in , or Figure 6 The image semantic segmentation model training method shown in , or Fig.11The image semantic segmentation method shown in . For example, the following image semantic segmentation method can be performed: repeatedly perform the step of extracting a feature map of a first image to be processed through the first branch and the second branch in the first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed that extracts the feature map through the first preset convolutional network for the first time is the original image, and the first feature map to be processed that is not the first time to extract the feature map through the first preset convolutional network is an image obtained by downsampling the feature map extracted by the first preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted each time through the first preset convolutional network; repeatedly perform the step of extracting the feature map of the first image to be processed for a second preset number of times. The steps of upsampling the second image to be processed and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting the feature map of the fused feature map through the third branch and the fourth branch in the second preset convolutional network, so as to obtain the target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network after the last upsampling, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels; determining the semantic segmentation result of the original image according to the target feature map.

[0135] In some embodiments of the present application, when the above program is running, it can also control the device where the non-volatile storage medium is located to execute the following image semantic segmentation model training method: obtain a training data set, the training data set includes sample images and label information corresponding to the sample images; input the training data set into the image semantic segmentation model to be trained, and obtain the predicted segmentation result output by the image semantic segmentation model to be trained, wherein the image semantic segmentation model to be trained includes an encoding module and a decoding module, the encoding module includes multiple first preset convolutional networks, the decoding module includes multiple second preset convolutional networks, the first preset convolutional network includes a first branch and a second branch, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes multiple hole convolution kernels, the second preset convolutional network includes a third branch and a fourth branch, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the second branch includes multiple hole convolution kernels; after each predicted segmentation result is obtained, compare the predicted segmentation result and the label information, and adjust the model parameters of the image semantic segmentation model to be trained according to the comparison result, until the comparison result meets the preset requirements; determine that the image semantic segmentation model to be trained with the parameters adjusted for the last time is the trained image semantic segmentation model.

[0136] In some embodiments of the present application, when the above program is running, it can also control the device where the non-volatile storage medium is located to perform the following image semantic segmentation method: obtain the original image; repeat the steps of extracting the feature map of the first image to be processed through the first branch and the second branch in the first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed that extracts the feature map through the first preset convolutional network for the first time is the original image, and the first image to be processed that does not extract the feature map through the first preset convolutional network for the first time is the image obtained by downsampling the feature map extracted by the last time through the preset convolutional network, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes multiple hole convolution kernels, and the feature map set includes the feature maps extracted each time through the first preset convolutional network. feature map; repeatedly performing the steps of upsampling the second image to be processed for a second preset number of times and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting the feature map of the fused feature map through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network in the last upsampling process, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes the first convolution kernel and the second convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels; determining the semantic segmentation result of the original image according to the target feature map.

[0137] It should be noted that, when the program is running, the device where the non-volatile storage medium is located can be controlled to execute the execution of the device where the non-volatile storage medium is located. Figure 2 The image semantic segmentation method shown in , or Figure 6 The image semantic segmentation model training method shown in , or Fig.11 The image semantic segmentation method shown in Figure 2 The image semantic segmentation method shown in , or Figure 6 The image semantic segmentation model training method shown in , or Fig.11 The relevant explanations and descriptions of the image semantic segmentation method shown in are also applicable to the above-mentioned non-volatile storage medium and will not be repeated here.

[0138] The embodiment of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to run a program stored in the memory, and executes Figure 2 The image semantic segmentation method shown in , or Figure 6 The image semantic segmentation model training method shown in , or Fig.11The image semantic segmentation method shown in . For example, the following image semantic segmentation method can be performed: repeatedly perform the step of extracting a feature map of a first image to be processed through the first branch and the second branch in the first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed that extracts the feature map through the first preset convolutional network for the first time is the original image, and the first feature map to be processed that is not the first time to extract the feature map through the first preset convolutional network is an image obtained by downsampling the feature map extracted by the first preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted each time through the first preset convolutional network; repeatedly perform the step of extracting the feature map of the first image to be processed for a second preset number of times. The steps of upsampling the second image to be processed and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting the feature map of the fused feature map through the third branch and the fourth branch in the second preset convolutional network, so as to obtain the target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network after the last upsampling, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels; determining the semantic segmentation result of the original image according to the target feature map.

[0139] In some embodiments of the present application, the following image semantic segmentation model training method can also be executed when the above program is running: obtain a training data set, the training data set includes sample images and label information corresponding to the sample images; input the training data set into the image semantic segmentation model to be trained, and obtain the predicted segmentation result output by the image semantic segmentation model to be trained, wherein the image semantic segmentation model to be trained includes an encoding module and a decoding module, the encoding module includes multiple first preset convolutional networks, the decoding module includes multiple second preset convolutional networks, the first preset convolutional network includes a first branch and a second branch, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes multiple hole convolution kernels, the second preset convolutional network includes a third branch and a fourth branch, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the second branch includes multiple hole convolution kernels; after each predicted segmentation result is obtained, compare the predicted segmentation result and the label information, and adjust the model parameters of the image semantic segmentation model to be trained according to the comparison result, until the comparison result meets the preset requirements; determine that the image semantic segmentation model to be trained with the parameters adjusted for the last time is the trained image semantic segmentation model.

[0140] In some embodiments of the present application, the following image semantic segmentation method can also be executed when the above program is running: obtaining an original image; repeatedly executing the step of extracting a feature map of the first image to be processed through the first branch and the second branch in the first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed that extracts the feature map through the first preset convolutional network for the first time is the original image, and the first image to be processed that does not extract the feature map through the first preset convolutional network for the first time is an image obtained by downsampling the feature map extracted by the preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes multiple hole convolution kernels, and the feature map set includes feature maps extracted each time through the first preset convolutional network; repeatedly executing A step of upsampling the second image to be processed a second preset number of times and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting a feature map of the fused feature map through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network during the last upsampling process, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a first convolution kernel and a second convolution kernel connected in series, and the fourth branch includes multiple hole convolution kernels; the semantic segmentation result of the original image is determined according to the target feature map.

[0141] It should be noted that when the program is running, the device where it is located can execute Figure 2 The image semantic segmentation method shown in , or Figure 6 The image semantic segmentation model training method shown in , or Fig.11 The image semantic segmentation method shown in Figure 2 The image semantic segmentation method shown in , or Figure 6 The image semantic segmentation model training method shown in , or Fig.11 The relevant explanations and descriptions of the image semantic segmentation method shown in are also applicable to the above-mentioned electronic devices and will not be repeated here.

[0142] The present application also provides a computer program product. The computer program product can be used to execute Figure 2 The image semantic segmentation method shown in , or Figure 6 The image semantic segmentation model training method shown in , or Fig.11 The image semantic segmentation method shown in Figure 2 The image semantic segmentation method shown in , or Figure 6 The image semantic segmentation model training method shown in , or Fig.11The relevant explanations and descriptions of the image semantic segmentation method shown in are also applicable to the computer program product and will not be repeated here.

[0143] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0148] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for image semantic segmentation, characterized in that: include: Repeat the step of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed for which a feature map is extracted through the first preset convolutional network for the first time is an original image, and the first feature map to be processed for which a feature map is not extracted through the first preset convolutional network for the first time is an image obtained by downsampling the feature map extracted through the first preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted through the first preset convolutional network each time; Repeat the steps of upsampling the second image to be processed for a second preset number of times and fusing it with the above feature map of the second image to be processed, and after each fusion, extract the feature map of the fused feature map obtained by fusion through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network after the last upsampling, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the fourth branch includes a plurality of hole convolution kernels; A semantic segmentation result of the original image is determined according to the target feature map.

2. The image semantic segmentation method according to claim 1, characterized in that: The first preset convolutional network also includes a fifth convolution kernel, and the step of extracting a feature map of the first image to be processed through the first branch and the second branch in the first preset convolutional network includes: Performing dimensionality reduction convolution processing on the first image to be processed by using the fifth convolution kernel to obtain a first intermediate feature map; Continuously convolving the first intermediate feature map with the first convolution kernel and the second convolution kernel in the first branch to obtain a first convolution result; Perform multiple dilated convolutions on the first intermediate feature map using the multiple dilated convolution kernels in the second branch to obtain a second convolution result; The first image to be processed, the first convolution result and the second convolution result are fused to obtain a feature map corresponding to the down-sampled image.

3. The image semantic segmentation method according to claim 2, characterized in that: The second branch includes a first dilated convolution kernel, a second dilated convolution kernel and a third dilated convolution kernel; the step of performing multiple dilated convolutions on the downsampled image using the multiple dilated convolution kernels in the second branch to obtain a second convolution result includes: Performing a first dilated convolution process on the first intermediate feature map using a first dilated convolution kernel to obtain a first dilated convolution result; Using a second dilated convolution kernel to perform convolution processing on the first intermediate feature map, and combining the first dilated convolution results to obtain a second dilated convolution result; The first intermediate feature map is convolved with a third dilated convolution kernel, and the second convolution result is merged to obtain a third dilated convolution result, wherein the second convolution result includes the first dilated convolution result, the second dilated convolution result and the third dilated convolution result.

4. The image semantic segmentation method according to claim 2, characterized in that: The step of fusing the first image to be processed, the first convolution result and the second convolution result to obtain a feature map corresponding to the image obtained after downsampling includes: Performing channel splicing processing on the first convolution result and the second convolution result to obtain a second intermediate feature map; Determining a dependency relationship between channels in the second intermediate feature map, and adjusting a feature response value of each channel according to the dependency relationship; The second intermediate feature map after adjusting the feature response value and the first image to be processed are fused to obtain a feature map corresponding to the first image to be processed.

5. The image semantic segmentation method according to claim 1, characterized in that: The upsampling of the second image to be processed and fusing the second image to be processed with the above feature map comprises: Determine the region of interest in the feature map above by a spatial attention network; According to the region of interest in the above feature map, the fusion weights of each channel in the above feature map are adjusted.

6. The image semantic segmentation method according to claim 1, characterized in that: The second preset convolutional network also includes a sixth convolution kernel, and the step of extracting a feature map of the fused feature map obtained by fusion through the third branch and the fourth branch in the second preset convolutional network includes: Performing dimensionality reduction convolution processing on the fused feature map through the sixth convolution kernel to obtain a third intermediate feature map; Continuously convolving the third intermediate feature map with the second convolution kernel and the third convolution kernel in the third branch to obtain a third convolution result; Performing multiple dilated convolutions on the third intermediate feature map through the multiple dilated convolution kernels in the fourth branch to obtain a fourth convolution result; The fused feature map, the third convolution result and the fourth convolution result are combined to obtain a feature map corresponding to the fused feature map.

7. The image semantic segmentation method according to claim 6, characterized in that: The fourth branch includes a fourth dilated convolution kernel, a fifth dilated convolution kernel and a sixth dilated convolution kernel; the step of performing multiple dilated convolutions on the downsampled image through the multiple dilated convolution kernels in the fourth branch to obtain a fourth convolution result includes: Using a fourth dilated convolution kernel to perform a first dilated convolution process on the third intermediate feature map to obtain a fourth dilated convolution result; Using a fifth dilated convolution kernel to perform convolution processing on the third intermediate feature map, and combining the fourth dilated convolution result to obtain a fifth dilated convolution result; The third intermediate feature map is convolved with the sixth dilated convolution kernel, and the fifth dilated convolution result is merged to obtain a sixth dilated convolution result, wherein the fourth convolution result includes the fourth dilated convolution result, the fifth dilated convolution result and the sixth dilated convolution result.

8. The image semantic segmentation method according to claim 6, characterized in that: The step of merging the fused feature map, the third convolution result and the fourth convolution result to obtain a feature map corresponding to the fused feature map includes: Performing channel splicing processing on the third convolution result and the fourth convolution result to obtain a fourth intermediate feature map; Determining a dependency relationship between channels in the fourth intermediate feature map, and adjusting a feature response value of each channel according to the dependency relationship; The fourth intermediate feature map after adjusting the feature response value and the fused feature map are fused to obtain a feature map corresponding to the fused feature map.

9. The image semantic segmentation method according to claim 1, characterized in that: The feature map set includes a shallow feature map, a middle feature map, a deep feature map and a bottom feature map, wherein the resolution of the shallow feature map is equal to the resolution of the image to be processed, the resolution of the middle feature map is smaller than the resolution of the image to be processed, the resolution of the deep feature map is smaller than the resolution of the middle feature map, and the resolution of the bottom feature map is smaller than the resolution of the deep feature map.

10. A method for training an image semantic segmentation model, characterized in that: include: Acquire a training data set, wherein the training data set includes sample images and label information corresponding to the sample images; Input the training data set into the image semantic segmentation model to be trained, and obtain the predicted segmentation result output by the image semantic segmentation model to be trained, wherein the image semantic segmentation model to be trained includes an encoding module and a decoding module, the encoding module includes a plurality of first preset convolutional networks, the decoding module includes a plurality of second preset convolutional networks, the first preset convolutional network includes a first branch and a second branch, the first branch includes a first convolutional kernel and a second convolutional kernel connected in series, the second branch includes a plurality of hole convolutional kernels, the second preset convolutional network includes a third branch and a fourth branch, the third branch includes a third convolutional kernel and a fourth convolutional kernel connected in series, and the second branch includes a plurality of hole convolutional kernels; After obtaining the predicted segmentation result each time, comparing the predicted segmentation result with the label information, and adjusting the model parameters of the image semantic segmentation model to be trained according to the comparison result until the comparison result meets the preset requirements; Determine that the image semantic segmentation model to be trained whose parameters are adjusted for the last time is the trained image semantic segmentation model.

11. The image semantic segmentation model training method according to claim 10, characterized in that: The image semantic segmentation model to be trained also includes a spatial attention module, a feature fusion module, an upsampling module, and a downsampling module, wherein: The spatial attention module is used to determine a region of interest in the image input to the spatial attention module, and adjust the fusion weights of the channels in the image input to the spatial attention module according to the region of interest; The feature fusion module is used to fuse the images input to the feature fusion module; The up-sampling module is used to perform up-sampling processing on the image input to the up-sampling module; The down-sampling module is used to perform down-sampling processing on the image input to the down-sampling module.

12. The image semantic segmentation model training method according to claim 10, characterized in that: The first preset convolution network also includes a first dimensionality reduction convolution kernel and a first splicing module, and the second convolution network also includes a second dimensionality reduction convolution kernel and a second splicing module, wherein: The first dimensionality reduction convolution kernel is used to perform dimensionality reduction convolution processing on the image input into the first convolution network; the second dimensionality reduction convolution kernel is used to perform dimensionality reduction convolution processing on the image input into the second convolution network; The first splicing module is used to fuse the result output by the first branch, the results output by each hole branch in the second branch, and the image input to the first preset convolutional network to obtain a feature map output by the first preset convolutional network; The second splicing module is used to fuse the result of the output of the third branch, the results of the output of each hole branch in the fourth branch, and the image input to the second preset convolutional network to obtain a feature map output by the second preset convolutional network.

13. The image semantic segmentation model training method according to claim 12, characterized in that: The first stitching module and the second stitching module also include a channel attention module, which is used to adjust the response characteristic values ​​of the channels in the image input to the channel attention module according to the dependencies between the channels in the image input to the channel attention module.

14. The image semantic segmentation model training method according to claim 12, characterized in that: The second branch includes a first hole branch, a second hole branch, and a third hole branch, wherein the first hole branch is used to perform hole convolution processing on the image output by the first dimensionality reduction convolution kernel and output it, the second hole branch is used to perform hole convolution processing on the image output by the first dimensionality reduction convolution kernel, and output the processed image after fusion with the output result of the first hole branch, and the third hole branch is used to perform hole convolution processing on the image output by the first dimensionality reduction convolution kernel, and output the processed image after fusion with the output result of the second hole branch; The fourth branch includes a fourth void branch, a fifth void branch, and a sixth void branch, wherein the fourth void branch is used to perform void convolution processing on the image output by the second dimensionality reduction convolution kernel and output it, the fifth void branch is used to perform void convolution processing on the image output by the second dimensionality reduction convolution kernel, and fuse the processed image with the output result of the fourth void branch and output it, and the sixth void branch is used to perform void convolution processing on the image output by the second dimensionality reduction convolution kernel, and fuse the processed image with the output result of the fifth void branch and output it.

15. A method for image semantic segmentation, characterized in that: include: Get the original image; Repeat the step of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed for which a feature map is extracted through the first preset convolutional network for the first time is an original image, and the first image to be processed for which a feature map is not extracted through the first preset convolutional network for the first time is an image obtained by downsampling the feature map extracted through the preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted through the first preset convolutional network each time; Repeat the steps of upsampling the second image to be processed for a second preset number of times and fusing it with the above feature map of the second image to be processed, and after each fusion, extract the feature map of the fused feature map obtained by fusion through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is the feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is the feature map extracted by the second preset convolutional network in the last upsampling process, the target feature map is the feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a first convolution kernel and a second convolution kernel connected in series, and the fourth branch includes a plurality of hole convolution kernels; A semantic segmentation result of the original image is determined according to the target feature map.

16. An image semantic segmentation device, characterized in that: include: A first processing module, used for repeatedly executing the step of extracting a feature map of a first image to be processed through a first branch and a second branch in a first preset convolutional network for a first preset number of times to obtain a feature map set, wherein the first image to be processed for which the feature map is extracted through the first preset convolutional network for the first time is an original image, and the first feature map to be processed for which the feature map is not extracted through the first preset convolutional network for the first time is an image obtained by downsampling the feature map extracted through the first preset convolutional network last time, the first branch includes a first convolution kernel and a second convolution kernel connected in series, the second branch includes a plurality of hole convolution kernels, and the feature map set includes feature maps extracted through the first preset convolutional network each time; A second processing module, used for repeatedly performing the steps of upsampling the second image to be processed for a second preset number of times and fusing it with the above feature map of the second image to be processed, and after each fusion, extracting a feature map of the fused feature map obtained by fusion through the third branch and the fourth branch in the second preset convolutional network, so as to obtain a target feature map, wherein the second image to be processed that is upsampled for the first time is a feature map with the smallest resolution in the feature map set, the second image to be processed that is not upsampled for the first time is a feature map extracted by the second preset convolutional network after the last upsampling, the target feature map is a feature map extracted by the second preset convolutional network after the last upsampling, the third branch includes a third convolution kernel and a fourth convolution kernel connected in series, and the fourth branch includes a plurality of hole convolution kernels; The third processing module is used to determine the semantic segmentation result of the original image according to the target feature map.

17. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the image semantic segmentation method described in any one of claims 1 to 9, or the image semantic segmentation model training method described in any one of claims 10-14, or the image semantic segmentation method described in claim 15.

18. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein when the program is run, the image semantic segmentation method described in any one of claims 1 to 9, or the image semantic segmentation model training method described in any one of claims 11 to 14, or the image semantic segmentation method described in claim 15 is executed.