Image semantic segmentation method and image semantic segmentation apparatus

Abstract

The application relates to the field of artificial intelligence, in particular to the field of computer vision, and provides an image semantic segmentation method and an image semantic segmentation device. The method comprises the following steps: acquiring an initial feature map of a to-be-processed image; inputting the initial feature map into a segmentation network, so as to obtain a semantic segmentation map of the to-be-processed image, the resolution of the semantic segmentation map being the same as that of the to-be-processed image; the segmentation network comprises at least one asymmetric convolution unit, the at least one asymmetric convolution unit is used for feature extraction on the initial feature map, and the shapes of a plurality of receptive fields generated by the asymmetric convolution unit are not completely the same. The single shape of the receptive field is expanded into the multi-shape receptive field, so that the shapes are more matched with actual objects and persons, and the segmentation precision is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to the field of computer vision, and particularly to an image semantic segmentation method and an image semantic segmentation device. Background Technology

[0002] Image semantic segmentation is a fundamental technique in computer vision. Its purpose is to divide an image into several specific regions with unique properties and identify the target region of interest. Image segmentation is a key step from image processing to image analysis. For example, in assisted driving scenarios, the output of semantic segmentation can help determine drivable areas, as well as vehicles and pedestrians on the road.

[0003] With the development and application of deep learning-based image segmentation technology in various fields, a variety of segmentation networks suitable for specific scenarios have emerged. Real-time urban street scenes are a crucial application scenario for image segmentation technology, playing a vital role in assisted driving and autonomous driving. In this application scenario, the segmentation task and the driving task are highly correlated. The results of semantic segmentation are used to assist autonomous driving systems in judging road conditions (semantic segmentation networks can label and separate pixels of roads, vehicles, and pedestrians, providing the driving system with drivable road surface location information). Therefore, high demands are placed on the accuracy and real-time performance of the segmentation network.

[0004] Urban street scene segmentation tasks typically run on automotive embedded platforms. Embedded platforms have limited computing power, memory, and power consumption. This application scenario requires a compact, efficient segmentation network with fast inference speed and minimal computational resources to complete the task, placing high demands on the lightweight nature of the segmentation network. High-precision, large-scale segmentation networks cannot meet this requirement, hence the emergence of lightweight semantic segmentation networks. The purpose of lightweight segmentation networks is to complete real-time semantic segmentation tasks under the constraints of embedded hardware platforms' computing power, memory, and other hardware limitations; therefore, the number of network parameters and computational cost must be sufficiently small.

[0005] Existing mainstream lightweight segmentation methods use depthwise separable convolution, Bottleneck structures, and asymmetric convolution to reduce the weight of segmentation models. However, the drawbacks of using these lightweight convolution operations are also very obvious: because these convolution operations simplify parameters or compress the channels of feature maps, the network's feature extraction ability is weak, which reduces the accuracy of the segmentation network.

[0006] Therefore, how to improve the accuracy of image semantic segmentation while ensuring a small amount of computation is an urgent technical problem to be solved. Summary of the Invention

[0007] This application provides an image semantic segmentation method and an image semantic segmentation apparatus, which can effectively improve segmentation accuracy while ensuring a small amount of computation.

[0008] In a first aspect, an image semantic segmentation method is provided, the method comprising: acquiring an initial feature map of an image to be processed, inputting the initial feature map into a segmentation network to obtain a semantic segmentation map of the image to be processed, wherein the resolution of the semantic segmentation map is the same as the resolution of the image to be processed, the segmentation network comprising at least one asymmetric convolutional unit, wherein the at least one asymmetric convolutional unit is used to extract features from the initial feature map, and the shapes of the multiple receptive fields generated by the asymmetric convolutional unit are not completely the same.

[0009] In the technical solution of this application, by expanding the receptive field of a single shape into a multi-shape receptive field, the receptive field is better matched with the shapes of real objects and people, effectively improving the segmentation accuracy. In addition, compared with symmetric convolution, the number of parameters and computational cost are significantly reduced by using asymmetric convolution, making it more suitable for lightweight model requirements, such as those required for high computational speed and segmentation accuracy in autonomous driving scenarios.

[0010] Alternatively, the image to be processed can be preprocessed to obtain an initial feature map, such as obtaining a three-channel initial feature map, or an already obtained initial feature map can be acquired.

[0011] The image to be processed can be any image, such as an image captured by a mobile phone, camera, or other terminal device, or an image captured or recorded by a camera or webcam in a vehicle. The image to be processed can be acquired in real time or read from a storage device such as a memory.

[0012] In some cases, the initial feature map can also be read from a storage device.

[0013] The segmentation network can adopt the framework structure of existing image semantic segmentation networks. The difference lies in that the segmentation network in this embodiment includes at least one asymmetric convolutional unit, which can generate multiple receptive fields with different shapes. That is, in addition to the aforementioned asymmetric convolutional unit, the segmentation network can still include common convolutional units or existing asymmetric convolutional units in the prior art. The difference is that these existing convolutional units, whether asymmetric or not, can only form square receptive fields and cannot form receptive fields of other shapes. This results in the receptive fields not being able to better match objects or people of different shapes, thus leading to lower segmentation accuracy.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the shape of the receptive field may include rectangles and squares.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the aforementioned shape is determined by the size and dilation rate of the convolution kernel of the asymmetric convolution unit. If symmetric convolution (e.g., 5×5 regular convolution or 3×3 dilated convolution) is used, then no matter how the size and dilation rate of the convolution kernel are adjusted, only a square receptive field can be obtained. If asymmetric convolution is used, then the asymmetric convolutions appear in groups (e.g., 1×5 regular convolution and 5×1 regular convolution, or 3×1 dilated convolution and 1×3 dilated convolution, etc.). When the dilation rate of the same group of asymmetric convolutions is the same, only a square receptive field can be obtained. Only when the dilation rates of the same group of asymmetric convolutions are different can a rectangular receptive field be obtained.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the aforementioned asymmetric convolutional unit further includes a point convolutional layer, which is used to compress the number of channels in the feature map input to the asymmetric convolutional unit.

[0017] The above segmentation network can be divided into shallow, medium and deep layers according to the relative relationship of the levels.

[0018] In some implementations, the above segmentation network can satisfy at least one of the following:

[0019] The shallow layers of the segmentation network include at least one shallow downsampling convolutional unit, which is used to extract shallow image feature information; or

[0020] The middle layer of the segmentation network includes at least one middle-layer downsampling convolutional unit, which is used to extract middle-layer image feature information; or

[0021] The deep layers of the segmentation network include at least one deep downsampled convolutional unit, which is used to extract deep semantic information.

[0022] The aforementioned shallow image feature information, mid-level image feature information, and deep semantic information are all processing results from different levels of the segmentation network. They are simply named according to the characteristics of the processing results to facilitate differentiation and understanding, and there are no other limitations.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the aforementioned shallow downsampling convolutional units, mid-level downsampling convolutional units, and deep downsampling convolutional units can also be set according to the hierarchical characteristics of the shallow, mid, and deep layers of the segmentation network.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the segmentation network described above may further include a feature fusion unit, which is used to fuse shallow image feature information, mid-level image feature information, and deep semantic information. This approach can effectively improve segmentation accuracy.

[0025] Since shallow layers have the highest image resolution and primarily extract image feature information, some implementations may omit asymmetric convolutional units in the shallow layers, instead using them only in the middle and deep layers. In other implementations, asymmetric convolutional units may be used only in the deep layers, while existing asymmetric convolutional units are used in the middle layers.

[0026] After multiple levels of asymmetric convolution and feature fusion, the resulting feature map has a lower resolution than the image to be processed. Therefore, upsampling can be used to adjust the fecal fraction to match that of the image. Common methods include upsampling convolution and bilinear upsampling. Bilinear upsampling can further reduce computation and parameter count, making it more suitable for lightweight semantic segmentation networks.

[0027] Secondly, a training method for an image semantic segmentation network is provided. The method includes: acquiring training data, which includes a training image and semantic segmentation labels corresponding to the training image; training a segmentation network using the training data to obtain a target segmentation network; wherein the training segmentation network can have any of the structures of the segmentation network described in the first aspect.

[0028] In the technical solution of this application, the segmentation network to be trained can have the same structure as the first aspect, and therefore can also have the same technical effects as the first aspect. For the sake of brevity, it will not be elaborated further.

[0029] In conjunction with the second aspect, some implementations of the second aspect involve data augmentation and data expansion of the training data to broaden its scope. For example, assuming the training image is a 1024*2048 resolution image, images with resolutions of 400*800 and 512*1024 can be obtained, and these three different resolution images can then be used as training data. This approach can more fully extract information from the training data, effectively making more efficient use of it, and, under otherwise unchanged conditions, can yield a more accurate model (network).

[0030] The training process can involve one or more iterations to update parameters such as the weights of the segmentation network to be trained. The number of training iterations or the training time can be controlled by setting a threshold for the number of training iterations, setting a training time, or setting the accuracy requirements for the target segmentation network, so as to output a target segmentation network that meets the above preset conditions.

[0031] The resulting target segmentation network can be used for image semantic segmentation, that is, it can be used as the segmentation network described in the first aspect.

[0032] Assuming that the training data, after processing, yields the three resolutions shown in the example above, then during training, the images at these three resolutions can be used sequentially from smallest to largest. This approach is equivalent to a coarse-to-fine training process, which can effectively improve training efficiency while keeping other conditions constant.

[0033] Thirdly, an image semantic segmentation apparatus is provided, the apparatus comprising various units for performing the method in the first aspect.

[0034] Fourthly, a training apparatus for an image semantic segmentation network is provided, the apparatus comprising units for performing the methods described in the second aspect above.

[0035] Fifthly, an image semantic segmentation apparatus is provided, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor executes the method of the first aspect.

[0036] In a sixth aspect, a training apparatus for an image semantic segmentation network is provided, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the second aspect above.

[0037] In a seventh aspect, an electronic device is provided, which includes the image semantic segmentation apparatus described in the third or fifth aspect above.

[0038] In the seventh aspect mentioned above, electronic devices may specifically be mobile terminals (e.g., smartphones), tablets, laptops, augmented reality / virtual reality devices, and in-vehicle terminal devices, etc.

[0039] Eighthly, a computer device is provided, which includes a training apparatus for the image semantic segmentation network described in the fourth or sixth aspect above.

[0040] In the eighth aspect mentioned above, computer equipment can specifically be servers or cloud devices, etc.

[0041] A ninth aspect provides a computer-readable medium storing program code for execution by a device, the program code including instructions for performing steps of the method in the first or second aspect.

[0042] In a tenth aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the methods described in the first or second aspect above.

[0043] Eleventhly, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the method of the first aspect or the second aspect.

[0044] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in the first aspect or the second aspect.

[0045] The aforementioned chip can be a field-programmable gate array or an application-specific integrated circuit.

[0046] It should be understood that, in this application, the method of the first aspect can specifically refer to the method of the first aspect and any one of the various implementations of the first aspect. The method of the second aspect can specifically refer to the method of the second aspect and any one of the various implementations of the second aspect. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the system architecture provided in the embodiments of this application.

[0048] Figure 2 This is a schematic diagram of semantic segmentation based on the FCN model provided in an embodiment of this application.

[0049] Figure 3 This is a schematic diagram of a chip hardware structure provided in an embodiment of this application.

[0050] Figure 4 This is a schematic structural diagram of an asymmetric convolution unit according to an embodiment of this application.

[0051] Figure 5 yes Figure 4 A schematic diagram of an example of the generated receptive field.

[0052] Figure 6 This is a schematic diagram of the segmentation network structure according to an embodiment of this application.

[0053] Figure 7 This is a schematic diagram illustrating an example of a segmentation network structure according to an embodiment of this application.

[0054] Figure 8 This is a schematic diagram of the structure of the downsampling convolution unit in an embodiment of this application.

[0055] Figure 9 yes Figure 7 A schematic diagram of an example of a segmentation network is shown.

[0056] Figure 10This is a schematic flowchart of an image semantic segmentation method according to an embodiment of this application.

[0057] Figure 11 This is a schematic flowchart of the training method of the image semantic segmentation network according to an embodiment of this application.

[0058] Figure 12 This is a schematic block diagram of an image semantic segmentation apparatus according to an embodiment of this application.

[0059] Figure 13 This is a schematic diagram of the hardware structure of the image semantic segmentation device provided in the embodiments of this application.

[0060] Figure 14 This is a schematic block diagram of a training apparatus for an image semantic segmentation network according to an embodiment of this application.

[0061] Figure 15 This is a schematic diagram of the hardware structure of the training device for the image semantic segmentation network provided in this application embodiment. Detailed Implementation

[0062] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0063] The image semantic segmentation scheme provided in this application can be applied to fields requiring semantic segmentation, such as autonomous driving systems, human-computer interaction systems, mobile device photography, image search engines, and augmented reality applications. A brief introduction is given below using an autonomous driving application scenario as an example.

[0064] Autonomous driving systems (ADS) utilize a variety of networks, including those for object detection, image recognition, semantic segmentation, and 3D pose estimation. These complex networks provide rich scene information, allowing the system to drive the car safely, replacing the driver.

[0065] Specifically, autonomous driving systems require semantic segmentation, which involves rapidly dividing the entire scene at a pixel-level precision. For example, one pixel might belong to a vehicle, while another might belong to a guardrail. Next, by converting pixel values ​​to physical distances, the system can complete a full model of the scene. Pixel-level positioning accuracy yields a relatively precise vehicle outline, enabling accurate calculation of lateral distances. Simultaneously, detected elements such as guardrails, lane lines, and road surfaces aid in determining the vehicle's more precise location.

[0066] Semantic segmentation can be achieved using the image semantic segmentation scheme of this application embodiment. Based on semantic segmentation, different objects in the image can be identified, thereby enabling the calculation of the vehicle's side distance and the determination of the vehicle's position.

[0067] It should be noted that the solution in this application embodiment can have both a traditional square receptive field and a rectangular receptive field, which is equivalent to expanding the single-shape receptive field into a multi-shape receptive field. This can better match people or objects of various shapes. Therefore, compared with the traditional solution that only has a square receptive field, it can better match objects of various shapes such as vehicles, people, roads, trees, and buildings in the image, thereby effectively improving the accuracy of semantic segmentation.

[0068] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0069] (1) Semantic segmentation

[0070] Unlike traditional image segmentation problems based on features such as grayscale, color, texture, and shape, semantic segmentation is typically pixel-level, meaning that each pixel in an image is labeled with a specific category. Semantic segmentation refers to not only segmenting objects into different categories in an image but also classifying those objects. Semantic segmentation usually takes an image as input and outputs a semantic segmentation map (prediction map), where each pixel contains the label of its corresponding category.

[0071] (2) Receptive field

[0072] In convolutional neural networks, the receptive field refers to the size of the region mapped on the original image by each pixel in the feature map output by each layer of the convolutional neural network. The original image here refers to the input image of the network (i.e., the feature map of the image), which can be understood as the image after preprocessing (such as resizing, warping, cropping).

[0073] A larger receptive field value indicates a wider range of original image that it can access, meaning it may contain more global and semantically higher-level features; conversely, a smaller value indicates that the features it contains are more local and detailed. Therefore, the value of the receptive field can be used to roughly determine the level of abstraction of each layer.

[0074] The size of the receptive field (i.e., scale) is affected by the number of convolutional layers, the convolutional kernel, the dilation rate, and the stride. Among these, the stride mainly affects the receptive field of the next convolutional layer, while the other parameters affect the receptive field of the current convolutional layer. That is, the size of the receptive field of the i-th convolutional layer is related to the number of layers, the convolutional kernel, the dilation rate, and the stride of the (i-1)-th layer, where i is a positive integer.

[0075] Furthermore, the effect of image edge padding is not considered when calculating the size of the receptive field.

[0076] The shape of the receptive field is mainly affected by the convolution kernel and the dilation rate. The shape and size (scale) of the receptive field are different concepts. In the embodiments of this application, two shape examples, square and rectangle, are mainly given.

[0077] (3) Neural Network

[0078] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit takes the intercept 1 as input, and its output can be as shown in formula (1):

[0079]

[0080] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, which performs a non-linear transformation on the features in the neural network, converting the input signal of the neural unit into an output signal. The output signal of this activation function can be used as the input to the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0081] (4) Deep Neural Networks

[0082] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0083] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also relatively large. The definitions of these parameters in DNNs are as follows: Taking the coefficient W as an example, assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0084] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0085] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0086] (5) Convolutional Neural Networks

[0087] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0088] (6) Residual Network

[0089] Residual networks, proposed in 2015, are a type of deep convolutional network. Compared to traditional convolutional neural networks, residual networks are easier to optimize and can improve accuracy by significantly increasing their depth. The core of residual networks is addressing the side effects (degradation) of increasing depth, thus improving network performance simply by increasing network depth. Residual networks typically contain many structurally identical sub-modules, usually connected by a number representing the number of times a sub-module is repeated; for example, ResNet50 indicates that the residual network has 50 sub-modules.

[0090] (7) Loss Function

[0091] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0092] (8) Backpropagation algorithm

[0093] Neural networks can employ backpropagation (BP) to correct the initial parameter values ​​during training, thereby minimizing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0094] The following is combined Figure 1 The system architecture of the embodiments of this application will be described in detail.

[0095] Figure 1This is a schematic diagram of the system architecture according to an embodiment of this application. For example... Figure 1 As shown, the system architecture 100 includes an execution device 110, a training device 120, a database 130, a client device 140, a data storage system 150, and a data acquisition system 160.

[0096] Additionally, the execution device 110 includes a calculation module 111, an I / O interface 112, a preprocessing module 113, and a preprocessing module 114. The calculation module 111 may include a target model / rule 101, while the preprocessing modules 113 and 114 are optional.

[0097] The data acquisition device 160 is used to acquire training data. For the image semantic segmentation scheme of this embodiment, the training data may include training images and corresponding label maps, wherein the label map provides the category label for each pixel in the training image. Further, the training data may also include inertial measurement unit (IMU) information corresponding to the training images. The IMU information may include pose information, such as the shooting angle information of the training images. The training data is stored in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0098] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 performs semantic segmentation on the input training image and compares the output semantic segmentation result (the predicted value of the category corresponding to each pixel in the image) with the label image until the difference between the semantic segmentation result output by the training device 120 and the label image is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0099] The aforementioned target model / rule 101 can be used to implement the image semantic segmentation method of this application embodiment. That is, by inputting the image to be processed into the target model / rule 101 (which may undergo relevant preprocessing before being input into the target model / rule 101), the semantic segmentation result of the image to be processed can be obtained. Specifically, the target model / rule 101 in this application embodiment can be a neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as a limitation of the embodiments of this application.

[0100] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 1 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 1 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data into the I / O interface 112 through the client device 140. In this embodiment, the input data may include an image to be processed input by the client device. Further, the input data may also include IMU information corresponding to the image to be processed. Specifically, the client device 140 may be a terminal device.

[0101] Preprocessing modules 113 and 114 are used to preprocess the input data (such as the image to be processed) received by I / O interface 112. In this embodiment, preprocessing modules 113 and 114 may be absent, or only one preprocessing module may be used. When preprocessing modules 113 and 114 are absent, the calculation module 111 can be used directly to process the input data.

[0102] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0103] Finally, I / O interface 112 presents the processing results, such as the semantic segmentation results calculated by the target model / rule 101, to client device 140, thereby providing them to the user.

[0104] Specifically, the semantic segmentation result obtained by the target model / rule 101 in the calculation module 111 can be processed by the preprocessing module 113 (or by the preprocessing module 114) and then sent to the I / O interface, and then the I / O interface sends the processing result to the client device 140 for display.

[0105] It should be understood that when the preprocessing module 113 and preprocessing module 114 are not present in the above system architecture 100, the computing module 111 can also transmit the processed semantic segmentation result to the I / O interface, and then the I / O interface sends the processing result to the client device 140 for display.

[0106] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0107] exist Figure 1 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0108] It is worth noting that, Figure 1 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 1 In this case, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0109] like Figure 1 As shown, the target model / rule 101 trained by the training device 120 can be a neural network as described in this application embodiment. Specifically, the neural network provided in this application embodiment can be a fully convolutional network (FCN), etc. In the neural network provided in this application embodiment, the segmentation network and its sub-networks (sub-modules) can all be fully convolutional networks.

[0110] As introduced in the basic concepts above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network, where each neuron can respond to the input image.

[0111] A convolutional neural network (CNN) can include an input layer, convolutional / pooling layers (where pooling layers are optional), and fully connected layers. Replacing fully connected layers with convolutional layers yields a fully convolutional network. Figure 2 This is a schematic diagram illustrating semantic segmentation based on the FCN model provided in an embodiment of this application. For example... Figure 2 As shown, a fully convolutional network may include an input layer 210, a convolutional / pooling layer 220 (where the pooling layer is optional), and a convolutional layer 230. The details of these layers are described below.

[0112] Convolutional / pooling layers 220:

[0113] like Figure 2 The convolutional / pooling layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0114] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0115] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During convolution, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. This depth can be understood as being determined by the "multiple" weight matrices mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix might be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these identical weight matrices also have the same size. These extracted feature maps are then merged to form the output of the convolution operation.

[0116] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the fully convolutional network 200 to make correct predictions.

[0117] When a fully convolutional network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the fully convolutional network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0118] Pooling layer:

[0119] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 2Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0120] Convolutional layer 230:

[0121] After processing by the convolutional / pooling layers 220, the fully convolutional network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the fully convolutional network 200 needs to utilize the convolutional layers 230 to generate one or a set of output heatmaps representing the required number of classes. Therefore, the convolutional layers 230 can include multiple convolutional layers (such as...). Figure 2 As shown in 231, 232 to 23n) and output layer 240, the parameters contained in this multi-layer convolutional layer can be pre-trained based on relevant training data for a specific task type, such as semantic segmentation, etc.

[0122] The final layer of the entire fully convolutional network 200 is the output layer 240, which has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. This loss function is applied after the entire fully convolutional network 200 has undergone forward propagation (e.g., ...). Figure 2 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 2 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the fully convolutional network 200 and the error between the output of the fully convolutional network 200 through the output layer and the ideal result.

[0123] It should be noted that, as Figure 2The fully convolutional network 200 shown is only an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models.

[0124] The following describes a chip hardware structure provided by an embodiment of this application.

[0125] Figure 3 This is a schematic diagram of a chip hardware structure provided in an embodiment of this application. The chip includes a neural network processor 30. The chip can be configured as follows: Figure 1 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 1 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. The algorithms for each layer in the convolutional neural network can be implemented in... Figure 3 This is achieved in the chip shown.

[0126] The neural network processing unit (NPU) is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned to it by the host CPU. The core of the NPU is the arithmetic circuit 303, which is controlled by the controller 304 to retrieve data from the memory (weight memory or input memory) and perform multiplication and addition operations.

[0127] In some implementations, the arithmetic circuit 303 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. The arithmetic circuit 303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0128] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 303 retrieves the weight data of matrix B from the weight memory 302 and caches it in each PE (Engineer Component) of the arithmetic circuit 303. The arithmetic circuit 303 retrieves the input data of matrix A from the input memory 301, performs matrix operations based on the input data of matrix A and the weight data of matrix B, and stores the partial or final result of the obtained matrix in the accumulator 308.

[0129] The vector computation unit 307 comprises multiple processing units that, when necessary, further process the output of the computation circuit 303, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. For example, the vector computation unit 307 can be used for computations in non-convolutional layers or fully connected layers (FC) of neural networks, such as pooling, batch normalization, and local response normalization.

[0130] In some implementations, vector computation unit 307 stores the processed vectors in unified memory 306. For example, vector computation unit 307 can apply nonlinear functions to the output of arithmetic circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, vector computation unit 307 generates normalized values, merged values, or both. In some implementations, the vectors processed by vector computation unit 307 can be used as activation inputs to arithmetic circuit 303, for example, for use in subsequent layers of a neural network.

[0131] The unified memory 306 is used to store input data and output data.

[0132] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the direct memory access controller 305 (DMAC), the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0133] The bus interface unit (BIU) 310 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 309 via a bus.

[0134] The instruction fetch buffer 309 connected to the controller 304 is used to store the instructions used by the controller 304;

[0135] The controller 304 is used to call the instructions cached in the instruction memory 309 to control the operation of the computing accelerator.

[0136] Generally, the unified memory 306, input memory 301, weighted memory 302, and instruction fetch memory 309 are all on-chip memories. External memory is independent of this NPU hardware architecture. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0137] In addition, the operations of each layer in the neural network can be performed by the operation circuit 303 or the vector calculation unit 307.

[0138] Figure 4 This is a schematic structural diagram of an asymmetric convolution unit according to an embodiment of this application. For example... Figure 4 As shown, the asymmetric convolutional unit 400 includes a receptive field convolutional group 410 and a feature fusion convolutional group 420.

[0139] The receptive field convolution group 410 includes four branches and one merging unit. The results from the four branches are merged in the merging unit. Figure 4 In this context, we use Concat to represent the branches. Each branch consists of a set of asymmetric convolutions, which will be described in detail below.

[0140] like Figure 4 As shown, the first branch includes a 3×1 regular convolution ( Figure 4 3×1 convolution and 1×3 regular convolution (in the context of the convolution) Figure 4 In the 1×3conv), and the dilation rate of both 3×1conv and 1×3conv is 1 ( Figure 4 (dilation = 1). It's important to note that dilated convolution with a dilation rate of 1 is equivalent to regular convolution. Dilated convolution can also be called dilated convolution, and the dilation rate can be called the dilation rate. The receptive field generated by the first branch is a square receptive field, and it is a relatively small-scale receptive field. It should also be understood that within the same convolutional layer, the scale and shape are affected by the size of the convolution kernel and the dilation rate. For example, a 3×1 convolution kernel and a 1×3 convolution kernel with a dilation rate of 1 will generate a receptive field with a scale of 3×3 and a square shape.

[0141] It should be noted that shape and scale are different concepts. Scale can be understood as size or dimensions, but shape is unrelated to size. In the embodiments of this application, shape specifically refers to shapes such as rectangles and squares, regardless of specific size. Other shapes may even appear in the future, such as trapezoids, parallelograms, and other shapes. In the prior art, the scale of the receptive field varies, but the shape of the receptive field is always a simple square.

[0142] Compared to symmetric convolution, asymmetric convolution effectively reduces computational load and improves operational efficiency while maintaining relatively less impact on accuracy, thus making it suitable for use in lightweight neural network models.

[0143] The second branch includes 3×1 dilated convolution ( Figure 4 The second branch of the middle branch uses 3×1dwconv and 1×3 dilated convolution. Figure 4 The second branch is 1×3dwconv), and the dilation rate of both 3×1dwconv and 1×3dwconv is dr( Figure 4 The second branch (dilation = dr) generates a square receptive field.

[0144] The third branch includes 3×1 dilated convolution ( Figure 4 The third branch of the 3×1dwconv and 1×3 dilated convolution ( Figure 4 The third branch of the 1×3dwconv, and the expansion rate of the 3×1dwconv is dr×2 ( Figure 4 The dilation of the third branch is dr×2, and the dilation rate of 1×3dwconv is dr×4. Figure 4 The dilation of the third branch is dr × 4. The third branch generates a rectangular receptive field.

[0145] The fourth branch includes 3×1 dilated convolution ( Figure 4 The fourth branch of the 3×1dwconv and 1×3 dilated convolution ( Figure 4 The fourth branch of the 1×3dwconv, and the expansion rate of the 3×1dwconv is dr×4 ( Figure 4 The dilation of the fourth branch is dr×4, and the dilation rate of 1×3dwconv is dr×2. Figure 4 The dilation of the fourth branch is dr × 2. The fourth branch generates a rectangular receptive field.

[0146] In some implementations, dimensionality reduction processing, such as point convolution, can be performed on the input data (feature map of the input branch) at each branch. Figure 4This shows that each branch consists of a 1×1 convolutional layer. However, it should be understood that the purpose of dimensionality reduction is to further reduce the computational cost, but the shape of the receptive field is not affected regardless of whether dimensionality reduction is performed.

[0147] It should be noted that, Figure 4 This only provides a specific example of an asymmetric convolutional unit. However, the size of the kernel within each branch and whether each branch includes a dotted base layer can be adjusted according to the actual situation, as long as multiple receptive fields can be formed, and the shapes of these receptive fields are not identical. For example, in the third and fourth branches, as long as the dilation rates of the two corresponding asymmetric convolutions within the same branch are different, rectangular receptive fields can be formed. However, there is no limitation on the specific value of the dilation rate, nor is there a limitation on the size of the convolution kernel. For a concrete example, the dilation rate of the 3×1dwconv in the third branch can be 5, and the dilation rate of the 1×3dwconv in the third branch can be any positive integer other than 5. As another example, the fourth branch can include 5×1dwconv and 1×5dwconv. That is to say, the kernel size of asymmetric convolutions does not necessarily have to be 3×1 and 1×3; there are other possibilities, which will not be listed here.

[0148] The feature fusion convolutional group 420 can be used to further process the feature map obtained from the receptive field convolutional group 410, such as feature fusion and / or feature transformation, thereby improving the information redundancy of the feature map and thus improving the segmentation accuracy. For example... Figure 5 As shown, the feature map output by the receptive field convolutional group 410 is merged with the feature map after it has undergone 3×3 dilation convolution, then superimposed after point convolution. The feature map after point convolution is then superimposed with the image input to the receptive field convolutional group 410 to obtain the output feature map. It should be understood that the feature fusion convolutional group 420 only needs to be used to improve the information redundancy of the feature map, but there are no restrictions on its specific structure. That is to say, the feature fusion convolutional group 420 can have any feasible structure. In practice, it may even be possible not to set up a feature fusion convolutional group 420.

[0149] In some implementations, the feature fusion convolutional group 420 may include a feature fusion layer (merging layer) to merge the processing results of different branches. It should be noted that while the feature fusion layer in an asymmetric convolutional unit is used to fuse information from different branches, the feature fusion unit in the segmentation network described below is used to fuse information from different layers of the neural network; both serve the same function (feature fusion).

[0150] In existing technologies, without asymmetric convolution (i.e., using symmetric convolution), only square receptive fields can be generated. For example, a 3×3 convolution kernel with a dilation rate of 1 generates a 3×3 square receptive field, while a dilation rate of 2 generates a 6×6 square receptive field. In other words, regardless of the dilation rate, only square receptive fields are generated. Existing technologies also use the same dilation rate for asymmetric convolution. For example, even if all kernels used are asymmetric (like the one in the first branch), they still generate square receptive fields. Even if the dilation rate changes, existing technologies use the same dilation rate within the same branch. For instance, the dilation rate of both the 3×1 convolution and the 1×3 convolution in the second branch is dr(…). Figure 4 If dilation = dr), then the generated receptive field is only different from the scale of the first branch, but the shape is still square.

[0151] Figure 5 yes Figure 4 A schematic diagram of an example of the generated receptive field. For example... Figure 4 As shown in (a), four receptive fields of different sizes and shapes are formed. From the inside out, the innermost square 501 corresponds to... Figure 4 The first branch forms the receptive field, and the square 502 in the second layer corresponds to... Figure 4 The receptive field formed by the second branch in the middle, the third layer of vertical rectangle 503 corresponds to Figure 5 The receptive field formed by the third branch in the middle, the fourth layer of horizontal rectangle 504 corresponds to Figure 5 The receptive field formed by the fourth branch of the middle limb. For example... Figure 5 As shown in (b), there are receptive fields of four shapes and four sizes, making it easier to match actual objects or pedestrians and achieving higher image segmentation accuracy, for example. Figure 5 As shown in (b), the vehicles there are a better match for the horizontal rectangle.

[0152] from Figure 5 It can be seen that the simultaneous existence of receptive fields of multiple sizes (multi-scale) and shapes allows for better matching of objects or people in the image, thus enabling more accurate image segmentation. Assuming only a single square field of view is used in existing technologies, even with multiple scales (i.e., different square sizes), the results for... Figure 6 The vehicle shown in (b) still cannot be matched better, which is equivalent to Figure 6 There are only two possible cases: square 501 and square 502.

[0153] Figure 6 This is a schematic diagram of the segmentation network structure according to an embodiment of this application. Figure 4As shown, the segmentation network includes a shallow convolutional module 610, a mid-level convolutional module 620, a deep convolutional module 630, and a merging module 640. The shallow convolutional module 610 is used to extract shallow information from the image, the mid-level convolutional module 620 is used to extract mid-level information from the image, and the deep convolutional module 630 is used to extract deep information from the image.

[0154] It should be understood that shallow, medium, and deep layers are three relative concepts, equivalent to different levels of a neural network. Therefore, the above module division is only a logical division based on network levels and does not involve the structure of each part. According to the characteristics of using neural networks to process images, the shallower the layer, the more image details (features) can be extracted, but the less semantic information; conversely, the deeper the layer, the less image details (features) can be extracted, but the more semantic information. Figure 6 The main idea is to combine the information extracted from different levels of the segmentation network, based on the hierarchical characteristics of the network, to achieve higher segmentation accuracy. To make it easy to understand the characteristics of the information obtained from different levels, the results of shallow feature extraction (i.e., shallow information) can be called shallow image feature information, the results of mid-level feature extraction (i.e., mid-level information) can be called mid-level image feature information, and the results of deep feature extraction (i.e., deep information) can be called deep semantic information.

[0155] In some implementations, shallow convolution can employ downsampling convolution, meaning that the shallow convolution module 610 uses downsampling convolution for feature extraction. It should be noted that the reason for downsampling in the shallow layers is that the shallow image (shallow feature map) has the highest resolution, and downsampling convolution can effectively reduce computational load and improve operational efficiency.

[0156] In some implementations, both the intermediate convolutional layer and the deep convolutional layer can include at least one asymmetric feature extraction. That is, both the intermediate convolutional module 620 and the deep convolutional module 630 can include at least one asymmetric convolutional module, for example, both can include at least one... Figure 7 The asymmetric convolution module shown.

[0157] Optionally, downsampling convolutions can also be included in both intermediate and deep convolutions. For example, at least one downsampling convolution can be set before or after the at least one asymmetric convolution module mentioned above. Adding downsampling convolutions in intermediate or deep layers can further reduce the computational load and improve operational efficiency.

[0158] In some implementations, different downsampling convolutions can be set according to different levels to adapt to the characteristics of different levels, thereby minimizing the amount of computation while minimizing the impact on segmentation accuracy.

[0159] Figure 7 The segmentation network shown extracts features at different levels, combines information from different levels, and finally uses point convolution to obtain a semantic segmentation map. This segmentation network extracts image features by stacking the asymmetric convolutional units mentioned above. In the shallow and middle layers of the segmentation network, specially designed downsampling convolutions are used to downsample the features, reducing the resolution of the shallow and middle layer feature maps to match the resolution of the deep layers, thus combining the detailed features of the shallow layers with the semantic features of the deep layers. The downsampling convolution structures of the shallow and middle layers are different, with the aim of preserving as much detail as possible in the shallow feature maps.

[0160] Figure 6 This is a schematic diagram illustrating an example of a segmentation network structure according to an embodiment of this application. Figure 7 It can be seen as Figure 7 The diagram shows a specific example of a segmented network structure. For example... Figure 7 As shown, the segmentation network includes downsampled convolution A( Figure 7 Downsampling A), asymmetric convolution unit, downsampling convolution B ( Figure 7 Downsampling (B) and downsampling convolution (C) Figure 7 Downsampling C) and feature fusion unit ( Figure 7 C with large circles in the middle) and point convolution ( Figure 8 (in 1×1conv). Figure 8 In this process, the shallow convolutional module includes downsampling convolution A and downsampling convolution B; the mid-level convolutional module includes downsampling convolution A, n1 asymmetric convolutional units, and downsampling convolution C; the deep convolutional module includes downsampling convolution A and n2 asymmetric convolutional units; and the merging module includes a feature fusion unit, which is used to fuse the results obtained from the shallow, mid, and deep layers, and transmit the fused result to the point convolutional layer. After point convolution, a semantic segmentation map can be generated. However, the resolution of the semantic segmentation map at this time is different from that of the original image to be processed. Upsampling is required to adjust the resolution to be the same as that of the original image to be processed.

[0161] from Figure 8 As can be seen, the image to be processed (the input image in the upper left corner) is processed into three paths after different levels of convolution. These three paths are then merged by a merging module and further processed by point convolution to obtain the semantic segmentation map (the output image in the lower right corner).

[0162] As mentioned above, downsampling convolutions can be configured differently based on the characteristics of different layers. The following section will combine... Figure 7 Let me introduce it. Figure 8 This is a schematic diagram of the structure of the downsampling convolution unit in an embodiment of this application.

[0163] Figure 8(a) in the middle is Figure 8 An example of a mid-downsampling convolution A, such as Figure 8 As shown in (a), the downsampling convolution A consists of two parallel operations, the first being a 3×3 regular convolution. Figure 8 (a) has a 3×3conv) with a step size of 2 ( Figure 8 (a) stride = 2), the second operation is 2×2 max pooling ( Figure 7 (a) uses 2×2 maxpooling with a stride of 2. This downsampling convolution also includes a merging unit ( Figure 8 (a) The circled C) is the merging unit used to merge the results of two operations.

[0164] Figure 8 (b) is Figure 8 An example of a mid-downsampling convolution B, such as Figure 8 As shown in (b), the downsampling convolution B consists of two sequential operations, the first of which is a 3×3 regular convolution. Figure 8 (b) 3×3conv), step size is 2 ( Figure 7 (b) stride = 2), the second operation is 3×3 dilated convolution ( Figure 8 (b) 3×3dwconv), with a step size of 2.

[0165] Figure 8 (c) in the middle is Figure 9 An example of a mid-downsampling convolution C, such as Figure 7 As shown in (c), the downsampling convolution C includes a 3×3 dilated convolution. Figure 9 The 3×3dwconv operation in (c) has a stride of 2. It should also be noted that the downsampling convolution C is mainly used for deeper layers, so depthwise separable convolution can be used. In depthwise separable convolution, the channel convolution and spatial convolution (convolution on height and width) are separated. Channel convolution is performed first, and then width and height convolution are performed on the channels.

[0166] The following example illustrates the structure of the segmentation network in this embodiment. For simplicity, the segmentation network will be referred to simply as the network, and the output of each layer of the segmentation network will be named the i-th layer feature map, where i represents the layer number of the network.

[0167] Figure 9 yes Figure 10 The diagram shows an example of a segmentation network. Below is a schematic diagram of... Figure 10 To introduce, Figure 4 The numbers on the left indicate the layer numbers of the network. However, it should be understood that the interlayer structure of the network is not a sequential stacking of layers, but needs to be considered in conjunction with the connection relationships.

[0168] The network's first and second layers are both downsampled convolutions A. The first layer's feature map has 16 channels and a resolution half that of the original image (either the image to be processed or the input image). The second layer's feature map has 64 channels and a resolution one-quarter that of the original image. The downsampled convolution A consists of a max-pooling convolution and a 3×3 convolution with a stride of 2, all connected in parallel.

[0169] Layers 3 through 8 of the network consist of six asymmetric convolutional units (n1 = 6 as mentioned above). The basic dilation rate (dr) of all six asymmetric receptive field convolutional units is 1. The input and output feature sizes of each asymmetric convolutional unit are the same. The resolution of the feature maps from layer 3 to layer 8 is 1 / 4 of the original image, and the number of channels is 64 for each layer. It should be noted that setting dr to 1 here is equivalent to forming square receptive fields in layers 3 through 8. In other words, this example demonstrates setting only square receptive fields in the middle layers of the network. However, it should be understood that asymmetric convolutional units with receptive fields of different shapes can be set in the middle and / or deeper layers of the network, depending on the actual situation.

[0170] The 9th layer of the network is a downsampled convolution A. The resolution of the feature map in the 9th layer is reduced to 1 / 8 of the original image, and the number of channels is increased to 128.

[0171] Layers 10 through 18 of the network consist of nine asymmetric convolutional units with different base dilation rates (i.e., n² = 9 as mentioned above), with dilation rates of 2, 4, 6, 2, 4, and 6 respectively, outputting feature maps from layers 10 to 18. It should be understood that the dilation rate settings here are merely an example and there are no numerical limitations. Layers 10 through 18 of the network generate multi-scale rectangular receptive fields.

[0172] In other words, layers 1 to 18 of the network are mainly used for encoding images. Layers 19 to 21 are the feature fusion part, which is used to fuse the results encoded by the previous layers.

[0173] The 19th layer of the network is a downsampled convolution B. The input to the 19th layer is the feature map of the 1st layer. The resolution of the feature map of the 19th layer is 1 / 8 of the original image, and the number of channels is 16. The downsampled convolution B consists of a 3×3 convolution with a stride of 2 and a 3×3 depthwise separable convolution with a stride of 2, cascaded together.

[0174] The 20th layer of the network is a downsampled convolution C. The input to the 20th layer is the feature map from the 8th layer. The resolution of the 20th layer feature map is 1 / 8 of the original image, and it has 64 channels. The downsampled convolution C consists of a 3×3 depthwise separable convolution with a stride of 2.

[0175] The 21st layer of the network is the merging unit layer (feature fusion unit or feature fusion layer). The input of the 21st layer is the feature map of the 18th layer, the 19th layer and the 20th layer. The 21st layer is mainly used to combine the input feature maps in the channel dimension. The resolution of the 21st layer feature map is 1 / 8 of the original image and the number of channels is 208.

[0176] The 22nd layer of the network is a point convolutional layer, which can compress the number of channels in the feature map of the 21st layer to the total number of categories. The output feature map of the 22nd layer is the prediction annotation map.

[0177] The 23rd layer of the network is an upsampling layer, which can use bilinear upsampling to restore the feature map of the 22nd layer to the same resolution as the original image.

[0178] As can be seen from the above examples, the asymmetric convolutional units with rectangular receptive fields can be mainly set in deep convolutional units, which can improve computational efficiency while ensuring segmentation accuracy.

[0179] Experiments show that the model trained by the segmentation network in the above example has only 0.35 megabytes (M) of parameters, and only 1.85 gigabytes (G) of computations on a single 360×640 resolution three-channel image, while achieving an image processing speed of 65 frames per second, meeting real-time requirements, and achieving a mean intersection-over-union (mIoU) ratio of 67.1%. In contrast, existing technologies mostly have gigabytes of parameters and require at least tens of gigabytes of computation (often expressed in GFLOPS). Although some lightweight segmentation networks exist, their segmentation accuracy is very low due to factors such as the simple receptive field shape, failing to meet practical needs, especially the high segmentation accuracy requirements of autonomous driving. Therefore, the solution in this application effectively improves the segmentation accuracy of lightweight segmentation networks.

[0180] Figure 7 This is a schematic flowchart of the image semantic segmentation method according to an embodiment of this application. The following is a description of... Figure 7 The process will be described step by step.

[0181] 1001. Obtain the initial feature map of the image to be processed.

[0182] Alternatively, the image to be processed can be preprocessed to obtain an initial feature map, such as obtaining a three-channel initial feature map, or an already obtained initial feature map can be acquired.

[0183] The image to be processed can be any image, such as an image captured by a mobile phone, camera, or other terminal device, or an image captured or recorded by a camera or webcam in a vehicle. The image to be processed can be acquired in real time or read from a storage device such as a memory.

[0184] In some cases, the initial feature map can also be read from a storage device.

[0185] 1002. Input the initial feature map into the segmentation network to obtain the semantic segmentation map of the image to be processed. The resolution of the semantic segmentation map is the same as the resolution of the image to be processed. The segmentation network includes at least one asymmetric convolutional unit. The at least one asymmetric convolutional unit is used to extract features from the initial feature map. The shapes of the multiple receptive fields generated by the asymmetric convolutional unit are not exactly the same.

[0186] The segmentation network can adopt the framework structure of existing image semantic segmentation networks. The difference lies in that the segmentation network in this embodiment includes at least one asymmetric convolutional unit, which can generate multiple receptive fields with different shapes. That is, in addition to the aforementioned asymmetric convolutional unit, the segmentation network can still include common convolutional units or existing asymmetric convolutional units in the prior art. The difference is that these existing convolutional units, whether asymmetric or not, can only form square receptive fields and cannot form receptive fields of other shapes. This results in the receptive fields not being able to better match objects or people of different shapes, thus leading to lower segmentation accuracy.

[0187] Optionally, the shape of the receptive field in the embodiments of this application may include a rectangle and a square.

[0188] The following explains why multi-shape receptive fields can improve segmentation accuracy. In real-world scenarios, taking roads as an example, suppose an image is obtained using a vehicle's sensing device. This image may include trees, vehicles, pedestrians, buildings, signs, green belts, etc. If there is only a square receptive field, even with multiple square receptive fields of different scales, these elements still cannot be segmented accurately, and the segmentation accuracy is greatly limited. However, if the solution of this application embodiment is adopted, it is equivalent to expanding the receptive field, using receptive fields of more shapes for semantic segmentation, which can better match objects and people of different shapes, thereby effectively improving segmentation accuracy.

[0189] In some implementations, the shape described above is determined by the size and dilation rate of the convolution kernel of the asymmetric convolution unit. If symmetric convolution (e.g., 5×5 regular convolution or 3×3 dilated convolution) is used, a square receptive field can only be obtained regardless of the kernel size and dilation rate. If asymmetric convolution is used, the asymmetric convolutions appear in groups (e.g., 1×5 regular convolution and 5×1 regular convolution, or 3×1 dilated convolution and 1×3 dilated convolution). When the dilation rates of the same group of asymmetric convolutions are the same, a square receptive field can still be obtained. Only when the dilation rates of the same group of asymmetric convolutions are different can a rectangular receptive field be obtained. Since relevant introductions and examples have already been given above, they will not be repeated here.

[0190] Under the same conditions, symmetric convolution has higher accuracy than asymmetric convolution. However, the computational cost and number of parameters of symmetric convolution are much greater than those of asymmetric convolution. Therefore, lightweight neural network models tend to use asymmetric convolution instead of symmetric convolution. The solution in this application is mainly applied to the construction of lightweight neural network models. Furthermore, current symmetric convolution cannot form receptive fields of shapes other than squares.

[0191] In some implementations, the aforementioned asymmetric convolutional unit further includes a point convolutional layer, which is used to compress the number of channels in the feature map input to the asymmetric convolutional unit. For details, please refer to... Figure 7 Introduction to related content.

[0192] This segmentation network can be divided into shallow, medium, and deep layers based on the relative relationship between the levels.

[0193] In some implementations, the above segmentation network can satisfy at least one of the following:

[0194] The shallow layers of the segmentation network include at least one shallow downsampling convolutional unit, which is used to extract shallow image feature information; or

[0195] The middle layer of the segmentation network includes at least one middle-layer downsampling convolutional unit, which is used to extract middle-layer image feature information; or

[0196] The deep layers of the segmentation network include at least one deep downsampled convolutional unit, which is used to extract deep semantic information.

[0197] As mentioned above, the shallow image feature information, mid-level image feature information, and deep semantic information are all processing results of different levels of the segmentation network. They are simply named according to the characteristics of the processing results to facilitate differentiation and understanding, and there are no other limitations.

[0198] Optionally, the aforementioned shallow downsampling convolutional units, mid-level downsampling convolutional units, and deep downsampling convolutional units can also be configured according to the hierarchical characteristics of the shallow, mid, and deep layers of the segmentation network. For example, as mentioned above... Figure 11 As shown, a shallow convolutional module includes downsampling A and downsampling B; that is, a combination of downsampling A and downsampling B can be used as an example of a shallow downsampling convolutional unit. (Above) Figure 11 As shown, the intermediate convolutional module includes downsampling A and downsampling C. In other words, a combination of downsampling A and downsampling C can serve as an example of an intermediate downsampling convolutional unit. (Above) Figure 12 As shown, the deep convolutional module includes downsampling A, which means that downsampling A can be used as an example of a deep downsampling convolutional unit.

[0199] In some implementations, the segmentation network described above may further include a feature fusion unit, which fuses shallow image features, mid-level image features, and deep semantic information. This approach effectively improves segmentation accuracy; details can be found above and will not be repeated here. The feature fusion unit may be included in the merging module described above.

[0200] Since shallow layers have the highest image resolution and primarily extract image feature information, some implementations may omit asymmetric convolutional units in the shallow layers, instead using them only in the middle and deep layers. In other implementations, asymmetric convolutional units may be used only in the deep layers, while existing asymmetric convolutional units are used in the middle layers.

[0201] After multiple levels of asymmetric convolution and feature fusion, the resulting feature map has a lower resolution than the image to be processed. Therefore, upsampling can be used to adjust the fecal fraction to match that of the image. Common methods include upsampling convolution and bilinear upsampling. Bilinear upsampling can further reduce computation and parameter count, making it more suitable for lightweight semantic segmentation networks.

[0202] Figure 12 This is a schematic flowchart illustrating the training method of the image semantic segmentation network according to an embodiment of this application. The following is a description of... Figure 6 Each step will be explained.

[0203] 1101. Obtain the data to be trained.

[0204] The training data includes the images to be trained and their corresponding semantic segmentation labels.

[0205] Optionally, data augmentation and data expansion can be performed on the training data to broaden its scope. For example, assuming the training image is a 1024*2048 resolution image, images with resolutions of 400*800 and 512*1024 can be obtained, and these three different resolution images can then be used as training data. This approach can more fully extract information from the training data, effectively making more efficient use of it, and, under otherwise identical conditions, can yield a more accurate model (network).

[0206] 1102. Use the training data to train the segmentation network to obtain the target segmentation network.

[0207] The segmentation network to be trained can have any of the structures of the segmentation networks described above, which will not be repeated for the sake of brevity.

[0208] The training process can involve one or more iterations to update parameters such as the weights of the segmentation network to be trained. The number of training iterations or the training time can be controlled by setting a threshold for the number of training iterations, setting a training time, or setting the accuracy requirements for the target segmentation network, so as to output a target segmentation network that meets the above preset conditions.

[0209] The obtained target segmentation network can be used for image semantic segmentation, that is, it can be used as a model for implementing the image semantic segmentation method or apparatus of the embodiments of this application.

[0210] Assuming that the training data at the three resolutions in the example above were obtained in step 1101, then during the training process, the images at the three resolutions can be used sequentially from smallest to largest. This method is equivalent to a coarse-to-fine training process, which can effectively improve training efficiency while keeping other conditions unchanged.

[0211] Figure 6 This is a schematic block diagram of an image semantic segmentation apparatus according to an embodiment of this application. Figure 7 The image semantic segmentation apparatus 2000 shown includes an acquisition unit 2001 and a processing unit 2002.

[0212] The acquisition unit 2001 and the processing unit 2002 can be used to execute the image semantic segmentation method of the present application embodiment. Specifically, the acquisition unit 2001 can execute the above step 1001, and the processing unit 2002 can execute the above step 1002.

[0213] The aforementioned processing unit 2002 can be divided into multiple modules according to different processing functions. Specifically, the processing unit 2002 may include... Figure 7The segmentation network shown includes a shallow convolutional module 610, a mid-level convolutional module 620, a deep convolutional module 630, and a merging module 640. The processing unit 2002 is capable of implementing... Figure 9 The processing unit 2002 may further include the functions of each module in the segmentation network shown. Figure 13 The segmentation network shown includes shallow convolutional modules, mid-level convolutional modules, deep convolutional modules, merging modules, and point convolutional modules, and may also include an upsampling module. The processing unit 2002 is capable of implementing... Figure 13 The processing unit 2002 may further include the functions of each module in the segmentation network shown. Figure 14 The segmentation network shown.

[0214] It should be understood that the processing unit 2002 in the above-described device 2000 may be equivalent to the processor 3002 in the device 3000 described below.

[0215] Figure 14 This is a schematic diagram of the hardware structure of the image semantic segmentation device provided in the embodiments of this application. Figure 15 The image semantic segmentation device 3000 shown (which can specifically be a computer device) includes a memory 3001, a processor 3002, a communication interface 3003, and a bus 3004. The memory 3001, processor 3002, and communication interface 3003 are interconnected via the bus 3004.

[0216] The memory 3001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 3001 may store a program. When the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 and the communication interface 3003 are used to execute the various steps of the image semantic segmentation method of the embodiments of this application.

[0217] The processor 3002 may be a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the image semantic segmentation apparatus of this application embodiment, or to execute the image semantic segmentation method of this application method embodiment.

[0218] The processor 3002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the image semantic segmentation method of this application can be completed by the integrated logic circuitry in the hardware of the processor 3002 or by instructions in software form. The processor 3002 can also be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 3001. The processor 3002 reads the information in the memory 3001 and, in conjunction with its hardware, performs the functions required by the units included in the image semantic segmentation apparatus of this application embodiment, or executes the image semantic segmentation method of this application method embodiment.

[0219] The communication interface 3003 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 3000 and other devices or communication networks. For example, the communication interface 3003 can be used to acquire an image to be processed or to acquire an initial feature map of the image to be processed.

[0220] Bus 3004 may include a pathway for transmitting information between various components of device 3000 (e.g., memory 3001, processor 3002, communication interface 3003).

[0221] Figure 15 This is a schematic block diagram of a training apparatus for an image semantic segmentation network according to an embodiment of this application. Figure 13 The training device 4000 for the image semantic segmentation network shown includes an acquisition unit 4001 and a processing unit 4002.

[0222] The acquisition unit 4001 and the processing unit 4002 can be used to execute the training method of the image semantic segmentation network of the present application embodiment. Specifically, the acquisition unit 4001 can execute the above step 1101, and the processing unit 4002 can execute the above step 1102.

[0223] It should be understood that the processing unit 4002 in the above-described device 4000 may be equivalent to the processor 5002 in the device 5000 described below.

[0224] Figure 15 This is a schematic diagram of the hardware structure of the training device for the image semantic segmentation network provided in this application embodiment. Figure 13 The training device 5000 for the image semantic segmentation network shown (specifically, the device 5000 can be a computer device) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, processor 5002, and communication interface 5003 are interconnected via the bus 5004.

[0225] The memory 5001 can be a ROM, a static storage device, a dynamic storage device, or RAM. The memory 5001 can store a program. When the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 and the communication interface 5003 are used to execute the various steps of the image semantic segmentation network training method of the embodiments of this application.

[0226] The processor 5002 may be a CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute related programs to implement the functions required by the units in the image semantic segmentation network training device of the present application embodiment, or to execute the image semantic segmentation network training method of the present application method embodiment.

[0227] The processor 5002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the training method for the image semantic segmentation network of this application can be completed by the integrated logic circuits in the hardware of the processor 5002 or by instructions in software form. The aforementioned processor 5002 can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 5001. The processor 5002 reads the information in the memory 5001 and, in conjunction with its hardware, performs the functions required by the units included in the training device of the image semantic segmentation network of this application embodiment, or executes the training method of the image semantic segmentation network of this application method embodiment.

[0228] The communication interface 5003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 5000 and other devices or communication networks. For example, the communication interface 5003 can be used to acquire an image to be processed or to acquire an initial feature map of the image to be processed.

[0229] Bus 5004 may include a pathway for transmitting information between various components of device 5000 (e.g., memory 5001, processor 5002, communication interface 5003).

[0230] It should be noted that, although Figure 15 The device 3000 shown ​ The illustrated device 5000 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, devices 3000 and 5000 also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 3000 and 5000 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that devices 3000 and 5000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... ​ , ​ All the devices shown.

[0231] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different devices to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0232] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0233] In the several embodiments provided in this application, it should be understood that the disclosed systems, methods, and apparatuses can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0234] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0235] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0236] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: a Universal Serial Bus flash disk (USB flash disk, UFD), which can also be simply referred to as a USB flash drive, portable hard drive, ROM, RAM, magnetic disk, or optical disk, and other media capable of storing program code.

[0237] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image semantic segmentation method, characterized in that, The method comprises: obtaining an initial feature map of a to-be-processed image; inputting the initial feature map into a segmentation network to obtain a semantic segmentation map of the to-be-processed image, the resolution of the semantic segmentation map being the same as that of the to-be-processed image, the segmentation network comprising at least two asymmetric convolution units, the at least two asymmetric convolution units being used for feature extraction on the initial feature map, the at least two asymmetric convolution units appearing in groups, shapes of a plurality of receptive fields generated by the asymmetric convolution units being not completely the same, the shapes comprising a rectangle and a square.

2. The method of claim 1, wherein, The shapes are determined by sizes and expansion rates of convolution kernels of the asymmetric convolution units.

3. The method of claim 1 or 2, wherein, The asymmetric convolution unit further comprises a point convolution layer, the point convolution layer being used for compressing a channel number of the initial feature map.

4. The method of claim 1 or 2, wherein, The segmentation network satisfies at least one of the following conditions: a shallow layer of the segmentation network comprises at least one shallow layer down-sampling convolution unit, the at least one shallow layer down-sampling convolution unit being used for extracting shallow layer image feature information; or a middle layer of the segmentation network comprises at least one middle layer down-sampling convolution unit, the at least one middle layer down-sampling convolution unit being used for extracting middle layer image feature information; or a deep layer of the segmentation network comprises at least one deep layer down-sampling convolution unit, the at least one deep layer down-sampling convolution unit being used for extracting deep layer semantic information.

5. The method of claim 4, wherein, The shallow layer down-sampling convolution unit, the middle layer down-sampling convolution unit and the deep layer down-sampling convolution unit are set according to hierarchical features of the shallow layer, the middle layer and the deep layer of the segmentation network.

6. The method of claim 4, wherein, The segmentation network further comprises a feature fusion unit, the feature fusion unit being used for fusing the shallow layer image feature information, the middle layer image feature information and the deep layer semantic information.

7. The method of claim 1 or 2, wherein, The at least two asymmetric convolution units are specifically located in the deep layer and / or the middle layer of the segmentation network.

8. The method of claim 1 or 2, wherein, The segmentation network further comprises a bilinear up-sampling unit, the bilinear up-sampling unit being used for obtaining the semantic segmentation map with the same resolution as that of the to-be-processed image.

9. An image semantic segmentation apparatus, characterized by comprising: The method comprises: an obtaining unit configured to obtain an initial feature map of a to-be-processed image; a processing unit configured to input the initial feature map into a segmentation network to obtain a semantic segmentation map of the to-be-processed image, the resolution of the semantic segmentation map being the same as that of the to-be-processed image, the segmentation network comprising at least two asymmetric convolution units, the at least two asymmetric convolution units appearing in groups, the at least two asymmetric convolution units being used for feature extraction on the initial feature map, shapes of a plurality of receptive fields generated by the asymmetric convolution units being not completely the same, the shapes comprising a rectangle and a square.

10. The apparatus of claim 9, wherein, The shapes are determined by sizes and expansion rates of convolution kernels of the asymmetric convolution units.

11. The apparatus of claim 9 or 10, wherein, The asymmetric convolution unit further comprises a point convolution layer, the point convolution layer being used for compressing a channel number of the initial feature map.

12. The apparatus of claim 9 or 10, wherein, The segmentation network satisfies at least one of the following conditions: a shallow layer of the segmentation network comprises at least one shallow layer down-sampling convolution unit, the at least one shallow layer down-sampling convolution unit being used for extracting shallow layer image feature information; or a middle layer of the segmentation network comprises at least one middle layer down-sampling convolution unit, the at least one middle layer down-sampling convolution unit being used for extracting middle layer image feature information; or a deep layer of the segmentation network comprises at least one deep layer down-sampling convolution unit, the at least one deep layer down-sampling convolution unit being used for extracting deep layer semantic information. The middle layer of the segmentation network comprises at least one middle-layer down-sampling convolution unit, which is configured to extract middle-layer image feature information. The deep layer of the segmentation network comprises at least one deep-layer down-sampling convolution unit, which is configured to extract deep-layer semantic information.

13. The apparatus of claim 12, wherein, The shallow-layer down-sampling convolution unit, the middle-layer down-sampling convolution unit and the deep-layer down-sampling convolution unit are configured according to the hierarchical characteristics of the shallow layer, the middle layer and the deep layer of the segmentation network.

14. The apparatus of claim 12, wherein, The segmentation network further comprises a feature fusion unit, which is configured to fuse the shallow-layer image feature information, the middle-layer image feature information and the deep-layer semantic information.

15. The apparatus of claim 9 or 10, wherein, The at least two asymmetric convolution units are located in the deep layer and / or the middle layer of the segmentation network.

16. The apparatus of claim 9 or 10, wherein, The segmentation network further comprises a bilinear up-sampling unit, which is configured to obtain the semantic segmentation map with the same resolution as the image to be processed.

17. An image semantic segmentation apparatus, characterized by comprising: Comprise: a memory for storing executable instructions; a processor for calling and running the executable instructions in the memory to execute the method of any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program instructions, and when the program instructions are run by the processor, the method of any one of claims 1 to 8 is implemented.

19. A computer program product, characterised in that, The computer program product comprises computer program code, and when the computer program code is run on the computer, the method of any one of claims 1 to 8 is implemented.

20. A chip, characterized by The chip comprises a processor and a data interface, and the processor reads instructions stored on the memory through the data interface to execute the method of any one of claims 1 to 8.

Images