Fish-eye image semantic segmentation method and device, computer device and storage medium

Abstract

The application relates to a fisheye image semantic segmentation method and device, computer equipment and a storage medium. The method comprises the following steps: inputting a fisheye image into a preset model to output a label result of each pixel in the fisheye image, the preset model being obtained by training based on sample fisheye images, each pixel in the sample fisheye images having been labeled; determining a preset effective field of view region according to the internal and external parameters of a shooting device and specific shooting parameters of the shooting device, the fisheye image being obtained based on the shooting device; and if a pixel located outside the preset effective field of view region in the fisheye image is labeled as being located inside the preset effective field of view region, labeling the pixel located outside the preset effective field of view region in the fisheye image as a background pixel. Since the semantic segmentation result of the fisheye image can be corrected, the pixel located outside the preset effective field of view region in the segmentation result is labeled as a background pixel, so that the segmentation accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, computer device, and storage medium for semantic segmentation of fisheye images. Background Technology

[0002] Image semantic segmentation is a crucial part of image processing and machine vision technology, focusing on image understanding, and is also an important branch of AI. Semantic segmentation involves classifying each pixel in an image, determining its category (e.g., background, person, or vehicle), and thus dividing the image into regions. Currently, semantic segmentation is widely used in scenarios such as autonomous driving and drone landing point determination.

[0003] Fisheye lenses, as ultra-wide-angle lenses, have a field of view of 180° or more, allowing for a very large shooting range. Therefore, they are widely used in video surveillance, virtual reality, 3D modeling, and visual navigation. In these applications, semantic segmentation of fisheye images is a common process. Related technologies typically achieve semantic segmentation by pixel classification. However, since these technologies classify the entire fisheye image pixel by pixel, and the entire fisheye image usually includes pixels outside the fisheye lens's field of view, these pixels may be misclassified. For example, a pixel that is clearly outside the fisheye lens's field of view might be identified as a pixel within the field of view and thus classified into a certain type, which is obviously incorrect, resulting in low semantic segmentation accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, and storage medium for semantic segmentation of fisheye images to address the aforementioned technical problems and the issue of low semantic segmentation accuracy.

[0005] A semantic segmentation method for fisheye images, the method comprising:

[0006] The fisheye image is input into the preset model, and the labeling result of each pixel in the fisheye image is output. The preset model is obtained by training on the sample fisheye image, and each pixel in the sample fisheye image has been labeled.

[0007] Based on the intrinsic and extrinsic parameters of the shooting equipment and the specific shooting parameters of the shooting equipment, the preset effective field of view area is determined, and the fisheye image is obtained based on the shooting equipment;

[0008] If pixels in the fisheye image that are outside the preset effective field of view are marked as being within the preset effective field of view, then the pixels in the fisheye image that are outside the preset effective field of view are marked as background pixels.

[0009] In one embodiment, the preset model further includes a fully connected layer and a spatial pyramid pooling layer; the spatial pyramid pooling layer is used to convert the feature vector output by the convolutional layer into a fixed-length feature vector and input it to the fully connected layer.

[0010] In one embodiment, the preset model includes an attention layer; wherein the feature corresponding to each pixel in the fisheye image is obtained by weighted aggregation of the features of pixels at other locations in the fisheye image through the attention layer.

[0011] In one embodiment, the preset model is obtained by compression based on a preset method or the preset model is reconstructed based on a backbone network. The backbone network is any one of the following models: MobileNetv2, MnasNet, FCN, HRNet, FastSCNN, and PSPNet. The preset method includes any one of the following methods: weight reduction, weight decomposition, data precision reduction, and weight sharing.

[0012] In one embodiment, the preset model further includes an upsampling layer, which is used to upsample the feature map generated after the fisheye image is input into the preset model.

[0013] In one embodiment, after inputting the fisheye image into a preset model and outputting the labeling result of each pixel in the fisheye image, the method further includes: correcting the labeling result of each pixel in the fisheye image based on a conditional random field.

[0014] A fisheye image semantic segmentation device, the device comprising:

[0015] The output module is used to input fisheye images into a preset model and output the labeling result of each pixel in the fisheye image. The preset model is obtained by training based on sample fisheye images, and each pixel in the sample fisheye image has been labeled.

[0016] The determination module is used to determine the preset effective field of view area based on the intrinsic and extrinsic parameters of the shooting device and the specific shooting parameters of the shooting device. The fisheye image is obtained based on the shooting device.

[0017] The marking module is used to mark pixels outside the preset effective field of view in a fisheye image as background pixels when pixels outside the preset effective field of view in the fisheye image are marked as being within the preset effective field of view.

[0018] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0019] The fisheye image is input into the preset model, and the labeling result of each pixel in the fisheye image is output. The preset model is obtained by training on the sample fisheye image, and each pixel in the sample fisheye image has been labeled.

[0020] Based on the intrinsic and extrinsic parameters of the shooting equipment and the specific shooting parameters of the shooting equipment, the preset effective field of view area is determined, and the fisheye image is obtained based on the shooting equipment;

[0021] If pixels in the fisheye image that are outside the preset effective field of view are marked as being within the preset effective field of view, then the pixels in the fisheye image that are outside the preset effective field of view are marked as background pixels.

[0022] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the following steps:

[0023] The fisheye image is input into the preset model, and the labeling result of each pixel in the fisheye image is output. The preset model is obtained by training on the sample fisheye image, and each pixel in the sample fisheye image has been labeled.

[0024] Based on the intrinsic and extrinsic parameters of the shooting equipment and the specific shooting parameters of the shooting equipment, the preset effective field of view area is determined, and the fisheye image is obtained based on the shooting equipment;

[0025] If pixels in the fisheye image that are outside the preset effective field of view are marked as being within the preset effective field of view, then the pixels in the fisheye image that are outside the preset effective field of view are marked as background pixels.

[0026] The aforementioned fisheye image semantic segmentation method, apparatus, computer equipment, and storage medium input a fisheye image into a preset model and output the labeling result of each pixel in the fisheye image. A preset effective field of view is determined based on the intrinsic and extrinsic parameters of the imaging device and its specific imaging parameters. If pixels in the fisheye image located outside the preset effective field of view are labeled as being within the preset effective field of view, then these pixels are labeled as background pixels. Because the semantic segmentation result of the fisheye image can be corrected by labeling pixels outside the preset effective field of view as background pixels, the segmentation accuracy can be improved. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the content of the original image in one embodiment;

[0028] Figure 2 This is a schematic diagram of the semantic segmentation of the original image in one embodiment;

[0029] Figure 3This is a flowchart illustrating a fisheye image semantic segmentation method in one embodiment;

[0030] Figure 4 This is an internal schematic diagram of a panoramic image in one embodiment;

[0031] Figure 5 This is a schematic diagram of a panoramic image converted into a fisheye image in one embodiment.

[0032] Figure 6 This is a schematic diagram illustrating the process of converting a panoramic image into a fisheye image in one embodiment.

[0033] Figure 7 This is a schematic diagram of the content of a fisheye image in one embodiment;

[0034] Figure 8 This is a schematic diagram of a convolution kernel in one embodiment;

[0035] Figure 9 This is a schematic diagram of a convolution kernel in another embodiment;

[0036] Figure 10 This is a schematic diagram of the convolution kernel in yet another embodiment;

[0037] Figure 11 This is a schematic diagram of a spatial pyramid pooling layer in one embodiment;

[0038] Figure 12 This is a schematic diagram of a spatial pyramid pooling layer in yet another embodiment;

[0039] Figure 13 This is a flowchart illustrating the fisheye image semantic segmentation method in another embodiment;

[0040] Figure 14 This is a structural block diagram of a fisheye image semantic segmentation device in one embodiment;

[0041] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] It is understood that the terms "first," "second," etc., used in this application may be used to describe various technical terms, but unless otherwise specified, these technical terms are not limited to these terms. These terms are only used to distinguish one technical term from another. For example, without departing from the scope of this application, the third preset threshold and the fourth preset threshold may be the same or different.

[0044] Image semantic segmentation is a crucial part of image processing and machine vision technology, focusing on image understanding, and is also an important branch of AI. Semantic segmentation involves classifying each pixel in an image, determining its category (e.g., background, person, or vehicle), and thus dividing the image into regions. Currently, semantic segmentation is widely used in scenarios such as autonomous driving and drone landing point determination.

[0045] Fisheye lenses, as ultra-wide-angle lenses, have a field of view of 180° or more, allowing them to capture a very large area. Therefore, they are widely used in video surveillance, virtual reality, 3D modeling, and visual navigation. In these applications, semantic segmentation of fisheye images is a common process. However, in related technologies, semantic segmentation of fisheye images may result in erroneous pixel segmentation, leading to lower semantic segmentation accuracy.

[0046] To address the problems existing in the aforementioned related technologies, embodiments of the present invention provide a fisheye image semantic segmentation method. This method can be applied to terminals, which may include, but are not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. It should be noted that the quantity "multiple" mentioned in the embodiments of this application refers to a quantity of "at least two," for example, "multiple" means "at least two."

[0047] Before describing the specific implementation methods of this application, let's first explain its main application scenarios. This application is mainly applied to image semantic segmentation scenarios, which, literally, means enabling the computer to segment images based on their semantics. For example, when inputting data into a computer... Figure 1 In this case, it can output Figure 2 In the field of image processing, semantics refers to the understanding of an image's content, which can be specifically demonstrated through pixel classification. For example, understanding the content of Image 1 semantically means three people riding three bicycles. Segmentation means separating different objects in an image from a pixel perspective, labeling each pixel in the original image, thus... Figure 2 The different color depths shown can be used to distinguish between a person and a bicycle.

[0048] In conjunction with the above embodiments, in one embodiment, see [link to embodiment]. Figure 3This paper presents a semantic segmentation method for fisheye images. Taking the application of this method to a terminal, with the terminal as the executing entity, as an example, the method includes the following steps:

[0049] 301. Input the fisheye image into the preset model and output the labeling result of each pixel in the fisheye image. The preset model is obtained after training based on the sample fisheye image, and each pixel in the sample fisheye image has been labeled.

[0050] 302. Based on the internal and external parameters of the shooting equipment and the specific shooting parameters of the shooting equipment, determine the preset effective field of view area. The fisheye image is obtained based on the shooting equipment.

[0051] 303. If there are pixels in the fisheye image that are outside the preset effective field of view area but are marked as being within the preset effective field of view area, then the pixels in the fisheye image that are outside the preset effective field of view area are marked as background pixels.

[0052] In step 301 above, the fisheye image can be an image captured by a fisheye lens, or it can be converted from a panoramic image; this embodiment of the invention does not specifically limit this. A fisheye lens is a lens with a focal length of 16mm or less and an angle of view close to or equal to 180°. It is an extreme wide-angle lens, and "fisheye lens" is its alternative name. To achieve the maximum photographic angle, the front lens element of this type of lens has a very short diameter and protrudes forward in a parabolic shape, quite similar to a fish's eye. A fisheye lens is a special type of ultra-wide-angle lens, and its angle of view must reach or exceed the range that the human eye can see. Therefore, there is a significant difference between the scene seen through a fisheye lens and the real world as seen by the human eye. The scenes that the human eye can see in real life have regular and fixed shapes, while the image produced by a fisheye lens goes beyond this scope.

[0053] When converting a panoramic image into a fisheye image, if the panoramic image is as follows: Figure 4 As shown, if the panoramic image has a height of H and a width of W, then converting it into a fisheye image is as follows: Figure 5 As shown. The radius r0 of the fisheye image is W / (2*π), and the coordinates of the center of the fisheye image are (C... x C y ), C x =C y =r0. The coordinates (x, y) of a pixel in the fisheye image. f ,y f The pixel's polar coordinates (r, θ) and its corresponding coordinates (x, θ) in the panoramic image. p ,y p The relationship between the two can be referenced by the following formula:

[0054]

[0055]

[0056] The calculation process for θ and r can be referred to the following formulas (3) and (4):

[0057]

[0058]

[0059] Using the above formula, any pixel in the panoramic image can be... Figure 4 The coordinates in the image are converted to those in the fisheye image. Figure 5 The coordinates in the image are used. Of course, in practice, the panoramic image can also be placed in a spherical coordinate system, and then, based on a pre-selected mapping angle, the pixels are mapped onto the sphere to obtain a fisheye image. For details, please refer to [reference needed]. Figure 6 .exist Figure 6 In the diagram, the rectangle formed by A1, D1, A2, and D2 represents the panoramic image. Point A with coordinates (x, y) represents a pixel in the panoramic image. By connecting point A to the center o of the sphere, point A can be projected onto the left hemisphere of the sphere. The position on the left hemisphere that is projected onto is related to the pre-selected mapping angle.

[0060] In step 301 above, the preset model is trained based on sample fisheye images. Considering that sample fisheye images may not be easy to acquire, in actual implementation, the sample panoramic image can be converted into a sample fisheye image in the same way as the panoramic image is converted into a fisheye image described above. For example, if a fisheye image is formed by sampling spherical mapping, then a single sample panoramic image can be converted into multiple sample fisheye images through different mapping angles. This method can significantly improve the efficiency of sample acquisition, and based on different mapping angles, the sample content can be enriched.

[0061] Furthermore, the preset model can be an encoder-decoder architecture, where both the encoder and decoder can employ convolutional neural network models; this embodiment of the invention does not impose specific limitations on this. Through the preset model, the labeling result of each pixel in the fisheye image can be output. That is, it can be determined which object each pixel in the fisheye image belongs to. Thus, different objects in the fisheye image can be distinguished. For example, the sky background in the fisheye image can be distinguished from the rest of the image; the sky background and the rest of the image are two different objects. As another example, different people in the fisheye image can be distinguished; different people are different objects. The statement that each pixel in the sample fisheye image has been labeled as an object in the sample fisheye image means that which pixels in the sample fisheye image belong to the same object has been determined; that is, which object each pixel belongs to is definite. For example, if there are 5 objects in a sample fisheye image, it can be determined that there are actually 5 types of pixels in the sample fisheye image. Each pixel is then labeled to represent which object in the sample fisheye image each pixel belongs to. This process is called the labeling process of the sample fisheye image.

[0062] like Figure 7 As shown, a fisheye image has a preset effective field of view, while the "black border" area around the edges of the fisheye image does not belong to the preset effective field of view but is considered part of the background area. In actual implementation, it is possible to classify pixels outside the preset effective field of view into the preset effective field of view, that is, to classify pixels in the background area as objects in the fisheye image. To address this, in step 302, the preset effective field of view can be determined first. The imaging device can directly capture a fisheye image or capture a normal image and then convert the normal image into a fisheye image according to the pixel mapping process described above. This embodiment of the invention does not specifically limit this. Taking a fisheye camera as an example, the intrinsic and extrinsic parameters of the imaging device can be the focal length and field of view, while the specific shooting parameters of the imaging device can be represented by the projection model set by the fisheye camera.

[0063] This invention does not specifically limit the method of determining the preset effective field of view area based on the intrinsic and extrinsic parameters of the shooting device and the specific shooting parameters of the shooting device. This includes, but is not limited to: if the shooting device is a fisheye camera, then the radius and center of the preset effective field of view area are calculated based on the focal length, field of view, and projection model set by the fisheye camera; the preset effective field of view area is determined based on the radius and center of the preset effective field of view area. For example, if the focal length of the shooting device is f, the field of view is 2θ, and the projection model of the fisheye camera is an equidistant projection model, then the radius r of the preset effective field of view area is r = f * θ, and the center of the preset effective field of view area is the projection point of the center of the shooting setup on the imaging plane.

[0064] In step 303, after obtaining the classification result of each pixel in the fisheye image, it can be determined whether there are pixels that are marked as objects in the fisheye image but are located outside the preset effective field of view. If so, it indicates that the pixel is an incorrectly segmented pixel, and thus the pixel can be corrected, that is, marked as a background pixel.

[0065] The method provided in this invention involves inputting a fisheye image into a preset model and outputting the labeling result of each pixel in the fisheye image. A preset effective field of view is determined based on the intrinsic and extrinsic parameters of the imaging device and its specific shooting parameters. If pixels in the fisheye image located outside the preset effective field of view are labeled as being within it, then these pixels are labeled as background pixels. Since the semantic segmentation result of the fisheye image can be corrected by labeling pixels outside the preset effective field of view as background pixels, segmentation accuracy can be improved.

[0066] In conjunction with the above embodiments, in one embodiment, the preset model includes a convolutional layer; wherein the convolutional kernel used in the convolutional layer is configured with holes based on a preset dilation rate.

[0067] As can be seen from the above embodiments, the preset model can be an encoder-decoder architecture, and both the encoder and decoder ends can adopt convolutional neural network models. Therefore, convolutional layers can be set within the models of both the encoder and decoder ends. In the field of image segmentation, when an image is input into a convolutional neural network model, it is usually convolved first and then pooled to reduce the image size while increasing the receptive field. However, since image segmentation prediction is a pixel-level output, it is necessary to upsample the smaller image size after pooling to the original image size for prediction. Common upsampling methods include bilinear interpolation, deconvolution (transposed convolution), upsampling, and up-pooling, etc., which are not limited in this invention.

[0068] The pooling operation described above allows for reference to a larger receptive field when predicting and classifying each pixel. As explained above, image semantic segmentation has two key aspects: pooling to reduce image size and increase the receptive field, and upsampling to expand the image size. However, some information is inevitably lost during the process of first reducing and then increasing the size. Therefore, in this embodiment of the invention, holes are added to the convolution kernel during the convolution operation, thereby enabling a larger receptive field without pooling.

[0069] like Figure 8 As shown, Figure 8 This indicates that a dilated convolution with a dilation rate of 1 generates a convolution kernel with a receptive field of 3x3. For example... Figure 9 As shown, Figure 9 This indicates the convolution kernel generated using dilated convolution with a dilation rate of 2. Figure 9 The receptive field of each pixel is equivalent to Figure 8 kernel size in the convolution, Figure 9 The receptive field of the convolution kernel is 7x7. For example... Figure 10 As shown, Figure 10 This indicates the convolution kernel generated using dilated convolution with a dilation rate of 3. Figure 10 The receptive field of each pixel is equivalent to Figure 9 kernel size in the convolution, Figure 9 The receptive field of the convolution kernel is 15*15.

[0070] The preset dilation rate defines the spacing between values ​​processed by the convolution kernel, i.e., the number of pixel intervals within the kernel. If the original kernel size is k, and the preset dilation rate is r, then the actual size of the dilated convolution kernel is z = k + 2*(k-1)*(r-1). For example, as... Figure 8 As shown, k=3 and r=1, therefore the actual size of the convolution kernel is 3*3. Figure 9 As shown, k=3 and r=2, therefore the actual size of the convolution kernel is 7*7. Figure 10 As shown, k=3 and r=4, so the actual size of the convolution kernel is 15*15.

[0071] The method provided in this invention, by using dilated convolution, can exponentially increase the receptive field without pooling loss, thus allowing each convolution output to contain a large range of information. Simultaneously, by increasing the receptive field without reducing image resolution, it can effectively combine the contextual information of surrounding pixels, resulting in more refined semantic segmentation.

[0072] In existing convolutional neural networks, for networks with a defined structure, a fixed-size image is required as input, such as 224*224, 32*32, or 96*96. Therefore, if the image to be used for semantic segmentation does not conform to this fixed size, a series of operations such as cropping or scaling are required, which reduces the accuracy of semantic segmentation. Based on the above problems and in conjunction with the above embodiments, in one embodiment, the preset model further includes a fully connected layer and a spatial pyramid pooling layer; the spatial pyramid pooling layer is used to convert the feature vectors output by the convolutional layers into fixed-length feature vectors and input them to the fully connected layer.

[0073] Convolutional neural networks (CNNs) require a fixed-size input image, primarily determined by the function of each layer within the CNN. A CNN mainly consists of convolutional layers, pooling layers, and fully connected layers. For convolutional layers, the size of the input image is not critical. For example, if the input image is 30*81 pixels and the convolutional kernel size is 5*5, a 26*77 pixel image can be obtained. In other words, the size of the input image does not affect the operation of the convolutional layer. Even with an input image size of 600*500 pixels, the convolutional layer can still function normally; that is, the convolutional layer has no requirement for the size of the input image, and convolution operations can be performed on images of any size.

[0074] For pooling operations, the pooling layer has no restrictions on the size of the input image. For example, the tree pooling is a 2x2 matrix. If the input image size is 30x40, it will result in a 15x20 image after pooling. If the input image size is 53x22, it will result in a 26x11 image after pooling. Therefore, the pooling layer has no restrictions on the size of the input image; pooling operations can be performed on images of any size.

[0075] As described above, convolutional neural networks mainly consist of convolutional layers, pooling layers, and fully connected layers. The convolutional and pooling layers do not have requirements regarding the size of the input image. However, the size W of the connection weight matrix in the fully connected layer becomes fixed after training. For example, if the input and output sizes from the convolutional layer to the fully connected layer are 50 and 30 neurons respectively, then the size of the connection weight matrix is ​​50*30. That is, the fully connected layer does have requirements regarding the size of the input image. Therefore, in this embodiment of the invention, a spatial pyramid pooling layer is added between the convolutional layer and the fully connected layer so that the output of the convolutional layer, after passing through the spatial pyramid pooling layer, can be transformed into a fixed-length feature vector.

[0076] To facilitate understanding, the algorithm process of spatial pyramid pooling is explained below:

[0077] like Figure 11 As shown, Figure 11 It is a simple two-layer network. Figure 11 The image below represents the input layer, which is an image of arbitrary size, assuming its size is (w, h). Figure 11 The top layer represents the output layer, which has 21 neurons. This means that the input is an image of arbitrary size, and the goal is to extract 21 features. The spatial pyramid feature extraction process is as follows:

[0078] like Figure 11 As shown, when an image is input, it can be divided into sections using scales of different sizes. Figure 11In this method, three different scale sizes were used to divide an input image, resulting in a total of 16 + 4 + 1 = 21 patches. From each of these 21 patches, a feature was extracted, which is the 21-dimensional feature vector to be extracted.

[0079] In the image on the left side of the output layer, the scale can be understood as 4, dividing a complete image into 16 tiles, with each tile having a size of (w / 4, h / 4). In the image in the middle of the output layer, the scale can be understood as 2, dividing a complete image into 4 tiles, with each tile having a size of (w / 2, h / 2). In the image on the left side of the output layer, the scale can be understood as 1, treating the entire image as a single tile, with a tile size of (w, h).

[0080] The spatial pyramid pooling process involves calculating the maximum value of each of the 21 patches, resulting in an output neuron for each patch. This transforms an image of arbitrary size into a fixed-size 21-dimensional feature vector. Of course, the 21-dimensional feature vector is just an example; in practice, other dimensions can be designed for the output, such as increasing the number of pyramid layers or changing the grid size. The three different scales described above can each be called a layer of the pyramid, and the size of each patch can be called the window size. If n*n features are needed to output from a specific layer of the pyramid, the window size can be set to (w / n, h / n) for pooling.

[0081] If a convolutional neural network has many layers, and the input is an image of arbitrary size, the network can continuously perform convolution and pooling until it reaches the last few layers, i.e., when it needs to connect to a fully connected layer. At this point, spatial pyramid pooling can be used to transform the image of arbitrary size into a feature vector of fixed size. The specific flowchart is as follows. Figure 12 As shown.

[0082] The method provided in this invention improves the model's generalization ability to handle multi-scale images by setting a fully connected layer and a spatial pyramid pooling layer between the fully connected layer and the convolutional layer. In other words, when performing semantic segmentation on images, there is no requirement for image size.

[0083] The basic idea of ​​attention mechanisms in computer vision is to enable the model to learn attention, that is, to allow the model to ignore irrelevant information and focus on key information. Based on this, semantic segmentation of an image, i.e., classifying a pixel in an image, also requires attention to the information of pixels surrounding that pixel. In one embodiment, the preset model includes an attention layer; wherein the features corresponding to each pixel in the fisheye image are obtained by weighted aggregation of the features of pixels at other locations in the fisheye image through the attention layer.

[0084] Attention mechanisms can be used to capture rich semantic information. Furthermore, the attention layer can be divided into a positional attention layer and a channel attention layer, which can be added to the tail of a convolutional neural network with dilated convolutional layers. In the positional attention layer, the feature update of any pixel in the image is performed by weighted aggregation of features at all locations in the image. The weights are determined by the similarity between features at two locations. That is, regardless of the distance between pixels, the higher the similarity between the features of pixels, the higher the weight of the pixel's features during weighted aggregation. Similarly, a similar self-attention mechanism is applied to the channel attention layer to learn the relationship between any two channel mappings, and a particular channel is updated by weighted summation of all channels.

[0085] Since discriminative feature representations are crucial for scene understanding, positional attention layers enhance the representational power of feature maps generated in convolutional neural networks by encoding a broader range of semantic information into the local receptive field. Each channel mapping of high-level semantic features can be viewed as a categorical semantic response, and different semantic responses are interconnected. By capturing the interdependencies between different channel mappings, the representational power of feature maps for specific semantics can be effectively enhanced, thus requiring the use of channel attention layers.

[0086] The method provided in this invention introduces an attention mechanism, which allows for the selection of focus locations, thereby generating more discriminative feature representations. Furthermore, the convolutional neural network consists of numerous attention layers, enabling the generation of attention-aware features, and the features of different layers adaptively change as the network deepens. Simultaneously, the attention layers can capture features at different locations in the image, thereby improving the accuracy of semantic segmentation.

[0087] Currently, in pursuit of classification accuracy, convolutional neural networks are becoming increasingly deep and complex, with deep residual networks reaching as many as 152 layers. However, in some real-world applications, such as mobile or embedded devices, such large and complex models are difficult to apply. First, the excessive size of the model leads to insufficient memory. Second, some scenarios require low latency or fast response times. For example, a slow response time in a pedestrian detection system for an autonomous vehicle could cause an accident. Therefore, small and efficient convolutional neural network models are crucial in these scenarios.

[0088] Therefore, the goal of designing convolutional neural network models is primarily to reduce model size while maintaining model performance and simultaneously improve model speed. Currently, there are two main approaches: one is to reduce model size while maintaining performance, and the other is to directly design and train a small model. Based on this, and in conjunction with the above embodiments, in one embodiment, the preset model is obtained by compression using a preset method, or the preset model is reconstructed from a backbone network. The backbone network can be any of the following models: MobileNetv2, MnasNet, FCN, HRNet, FastSCNN, and PSPNet; the preset method includes any of the following: weight reduction, weight decomposition, data precision reduction, and weight sharing.

[0089] In the first scenario, the preset model is obtained after compression. The preset method for compressing the preset model can be as described above. The reason for using weight reduction for model compression is that many weights in the neural network are almost zero, and these weights have little effect; removing some weights does not affect the model's prediction performance. The reason for using weight decomposition for model compression is that the weight matrix can be decomposed into a low-rank matrix, making some weights zero, which can then be removed. Additionally, data precision reduction refers to adjusting some data from float64 to float32, and from float to int, thereby reducing the model size. The reason for using weight sharing for model compression is that the weights of many layers in the model can be shared, eliminating the need for individually setting weights for each layer. Of course, besides the preset methods described above, other methods for compressing the model include convolution splitting, knowledge distillation, channel compression, reducing the number of layers, and reducing the model input size. This embodiment of the invention does not specifically limit these methods.

[0090] In the second scenario, the backbone network refers to a lightweight convolutional network. Taking MobileNetv2 as an example, we will explain MobileNet. The basic unit of MobileNet is the depthwise separable convolution. Depthwise separable convolution is essentially a factorized convolution operation, which can be broken down into two smaller operations: depthwise convolution and pointwise convolution. Depthwise convolution differs from standard convolution. Standard convolution uses kernels on all input channels, while depthwise convolution uses a different kernel for each input channel; that is, one kernel corresponds to one input channel.

[0091] Therefore, depthwise convolution is a depth-level operation. Pointwise convolution is essentially a regular convolution, but it uses a 1x1 kernel. For depthwise separable convolution, it first performs depthwise convolution on different input channels separately, and then uses pointwise convolution to combine the outputs. In this way, the overall effect is similar to a standard convolution, but it greatly reduces the amount of computation and the number of model parameters.

[0092] The method provided in this embodiment of the invention reduces the number of model parameters and the computational load of the model without significant loss of accuracy, thereby improving the processing speed of the model, since the preset model is obtained after compression or by directly designing and training a small model.

[0093] The main purposes of image downsampling (also known as subsampling or downsampling) are twofold: first, to make the image fit the size of the display area; and second, to generate a thumbnail of the corresponding image. Image upsampling (also known as upsampling or image interpolation) primarily aims to enlarge the original image so that it can be displayed on a higher resolution display device. Scaling an image does not add any new information about it, thus inevitably affecting image quality.

[0094] The main principle of downsampling is as follows: For an image of size M*N, downsampling by a factor of s yields an image with a resolution of (M / s)*(N / s), where s is the common divisor of M and N. If considering a matrix-like image, the image within an s*s window of the original image can be transformed into a single pixel, and the value of this pixel is the average of all pixels within the window. The main principle of upsampling is as follows: Using interpolation, new pixels are inserted between existing image pixels using a suitable interpolation algorithm.

[0095] To minimize the impact on image quality and thus the accuracy of semantic segmentation, in one embodiment, the preset model further includes an upsampling layer, which is used to upsample the feature map generated after the fisheye image is input into the preset model. There are many upsampling methods, such as nearest neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation. In this embodiment, transposed convolution can be used. Transposed convolution is often used in convolutional neural networks to represent a type of inverse convolution, but it is not a strictly mathematically defined deconvolution operation. Unlike the inverse operation of pooling (unpooling), using transposed convolution to upsample the image allows for parameter learning. Transposed convolution is typically used to upsample the results of convolutional layers, bringing them back to the resolution of the original image. It should be noted that if the device is incompatible with transposed convolution in actual implementation, upsampling can be performed using a different method than transposed convolution, and an additional convolutional layer can be added for parameter learning.

[0096] The method provided in this embodiment of the invention can restore the size of the feature map through the upsampling layer, and the samples used to train the preset model can also restore the size of the feature map generated in the middle of the preset model through upsampling, thereby enabling the classification of pixels in the image based on more information, and thus improving the segmentation accuracy.

[0097] In conjunction with the above embodiments, in one embodiment, after inputting the fisheye image into a preset model and outputting the labeling result of each pixel in the fisheye image, the method further includes: correcting the labeling result of each pixel in the fisheye image based on a conditional random field.

[0098] Conditional random fields (CRFs) are graphical models without direction, where pixels in an image represent random vectors, and the lines connecting pixels represent the dependencies between these random vectors. In a CRF, the random variable Y is distributed as conditional probability, and the given observed value is the random variable X.

[0099] Applying Conditional Random Fields (CRF) to image semantic segmentation can be approached by representing a fisheye image as an undirected graph. Each pixel is a vertex in this graph, and the connections between pixels are represented by lines connecting vertices. Assigning each vertex a different label (target or background) allows for the correct separation of edges at the graph's boundaries. In image segmentation, two pixels with similar location and color features are more likely to be assigned the same label, thus reducing the likelihood of them being segmented. This corresponds to the probabilistic model in CRF. CRF can globally normalize all features, enabling the search for a globally optimal solution. In the context of image semantic segmentation, this can be achieved by minimizing the energy function, thereby obtaining the global optimum.

[0100] The method provided in this embodiment of the invention can improve segmentation accuracy by correcting the labeling results of each pixel in the fisheye image based on a conditional random field.

[0101] In conjunction with the above embodiments, in one embodiment, see [link to embodiment]. Figure 13 After inputting the fisheye image into the preset model and outputting the labeled result of each pixel in the fisheye image, it also includes:

[0102] 1301. Based on the marking results of each pixel in the fisheye image, determine the region in the fisheye image where the object to be replaced is located;

[0103] 1302. Replace the object to be replaced in the fisheye image according to the area where the object to be replaced is located.

[0104] Taking the sky as an example, the area of ​​the sky in the fisheye image can be determined based on the marking results of each pixel. Then, the sky background in the fisheye image is replaced according to the area of ​​the sky.

[0105] The method provided in this invention determines the region of the object to be replaced in a fisheye image based on the marking results of each pixel in the fisheye image. The object to be replaced is then replaced in the fisheye image according to its region. Because object replacement is possible, the applicability of this preset model can be improved.

[0106] It should be understood that, although Figure 1 and Figure 13 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 and Figure 13At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0107] It should be noted that the technical solutions described above can be implemented as independent embodiments or combined with each other as combined embodiments in actual implementation. Furthermore, the description of the embodiments of the present invention is based solely on the convenience of explanation, and different embodiments are described in a corresponding order, such as according to the order of data flow, rather than limiting the execution order between different embodiments. Accordingly, in actual implementation, if multiple embodiments provided by the present invention need to be implemented, it is not necessary to follow the execution order provided in the description of the embodiments; instead, the execution order between different embodiments can be arranged according to requirements.

[0108] In conjunction with the above embodiments, in one embodiment, such as Figure 14 As shown, a fisheye image semantic segmentation device is provided, including: an output module 1301, a determination module 1302 and a labeling module 1303;

[0109] The output module 1301 is used to input the fisheye image into the preset model and output the labeling result of each pixel in the fisheye image. The preset model is obtained after training based on the sample fisheye image, and each pixel in the sample fisheye image has been labeled.

[0110] The determination module 1302 is used to determine the preset effective field of view area based on the internal and external parameters of the shooting device and the specific shooting parameters of the shooting device. The fisheye image is obtained based on the shooting device.

[0111] The marking module 1303 is used to mark the pixels in the fisheye image that are outside the preset effective field of view as background pixels when the pixels in the fisheye image that are outside the preset effective field of view are marked as being within the preset effective field of view.

[0112] In one embodiment, the preset model includes a convolutional layer; wherein the convolutional kernel used by the convolutional layer has holes set based on a preset dilation rate.

[0113] In one embodiment, the preset model further includes a fully connected layer and a spatial pyramid pooling layer; the spatial pyramid pooling layer is used to convert the feature vector output by the convolutional layer into a fixed-length feature vector and input it into the fully connected layer.

[0114] In one embodiment, the preset model includes an attention layer; wherein the feature corresponding to each pixel in the fisheye image is obtained by weighted aggregation of the features of pixels at other locations in the fisheye image through the attention layer.

[0115] In one embodiment, the preset model is obtained by compression based on a preset method or the preset model is reconstructed based on a backbone network. The backbone network is any one of the following models: MobileNetv2, MnasNet, FCN, HRNet, FastSCNN, and PSPNet. The preset method includes any one of the following methods: weight reduction, weight decomposition, data precision reduction, and weight sharing.

[0116] In one embodiment, for any frame of image, the preset model further includes an upsampling layer, which is used to upsample the feature map generated after the fisheye image is input into the preset model.

[0117] In one embodiment, the device further includes a correction module.

[0118] The correction module is used to correct the labeling results of each pixel in the fisheye image based on the conditional random field.

[0119] Specific limitations regarding the fisheye image semantic segmentation device can be found in the limitations of the fisheye image semantic segmentation method described above, and will not be repeated here. Each module in the aforementioned fisheye image semantic segmentation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0120] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 14As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a fisheye image semantic segmentation method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0121] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0122] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0123] The fisheye image is input into the preset model, and the labeling result of each pixel in the fisheye image is output. The preset model is obtained by training on the sample fisheye image, and each pixel in the sample fisheye image has been labeled.

[0124] Based on the intrinsic and extrinsic parameters of the shooting equipment and the specific shooting parameters of the shooting equipment, the preset effective field of view area is determined, and the fisheye image is obtained based on the shooting equipment;

[0125] If pixels in the fisheye image that are outside the preset effective field of view are marked as being within the preset effective field of view, then the pixels in the fisheye image that are outside the preset effective field of view are marked as background pixels.

[0126] In one embodiment, when the processor executes a computer program, the preset model includes a convolutional layer; wherein the convolutional kernel used by the convolutional layer is configured with holes based on a preset dilation rate.

[0127] In one embodiment, when the processor executes a computer program, the preset model further includes a fully connected layer and a spatial pyramid pooling layer; the spatial pyramid pooling layer is used to convert the feature vectors output by the convolutional layer into feature vectors of fixed length and input them into the fully connected layer.

[0128] In one embodiment, when the processor executes a computer program, the preset model includes an attention layer; wherein the feature corresponding to each pixel in the fisheye image is obtained by weighted aggregation of the features of pixels at other locations in the fisheye image through the attention layer.

[0129] In one embodiment, when the processor executes the computer program, the preset model is obtained by compression based on a preset method or the preset model is reconstructed based on the backbone network. The backbone network is any one of the following models: MobileNetv2, MnasNet, FCN, HRNet, FastSCNN, and PSPNet. The preset method includes any one of the following methods: weight reduction, weight decomposition, data precision reduction, and weight sharing.

[0130] In one embodiment, when the processor executes a computer program, the preset model further includes an upsampling layer, which is used to upsample the feature map generated after the fisheye image is input into the preset model.

[0131] In one embodiment, after inputting a fisheye image into a preset model and outputting the labeling result of each pixel in the fisheye image, the processor further performs the following steps when executing the computer program:

[0132] Based on conditional random fields, the labeling results of each pixel in the fisheye image are corrected.

[0133] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0134] The fisheye image is input into the preset model, and the labeling result of each pixel in the fisheye image is output. The preset model is obtained by training on the sample fisheye image, and each pixel in the sample fisheye image has been labeled.

[0135] Based on the intrinsic and extrinsic parameters of the shooting equipment and the specific shooting parameters of the shooting equipment, the preset effective field of view area is determined, and the fisheye image is obtained based on the shooting equipment;

[0136] If pixels in the fisheye image that are outside the preset effective field of view are marked as being within the preset effective field of view, then the pixels in the fisheye image that are outside the preset effective field of view are marked as background pixels.

[0137] In one embodiment, when a computer program is executed by a processor, a preset model includes convolutional layers; wherein the convolutional kernels used in the convolutional layers are configured with holes based on a preset dilation rate.

[0138] In one embodiment, when the computer program is executed by the processor, the preset model further includes a fully connected layer and a spatial pyramid pooling layer; the spatial pyramid pooling layer is used to convert the feature vectors output by the convolutional layer into feature vectors of fixed length and input them into the fully connected layer.

[0139] In one embodiment, when the computer program is executed by the processor, the preset model includes an attention layer; wherein the feature corresponding to each pixel in the fisheye image is obtained by weighted aggregation of the features of pixels at other locations in the fisheye image through the attention layer.

[0140] In one embodiment, when the computer program is executed by the processor, the preset model is obtained by compression based on a preset method or the preset model is reconstructed based on a backbone network. The backbone network is any one of the following models: MobileNetv2, MnasNet, FCN, HRNet, FastSCNN, and PSPNet. The preset method includes any one of the following methods: weight reduction, weight decomposition, data precision reduction, and weight sharing.

[0141] In one embodiment, when the computer program is executed by the processor, the preset model further includes an upsampling layer, which is used to upsample the feature map generated after the fisheye image is input into the preset model.

[0142] In one embodiment, after inputting a fisheye image into a preset model and outputting the labeling result of each pixel in the fisheye image, the computer program, when executed by a processor, further performs the following steps:

[0143] Based on conditional random fields, the labeling results of each pixel in the fisheye image are corrected.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0145] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A semantic segmentation method for fisheye images, characterized in that, The method includes: A fisheye image is input into a preset model, and the labeling result of each pixel in the fisheye image is output. The preset model is obtained by training on sample fisheye images, and each pixel in the sample fisheye image has been labeled. Based on the intrinsic and extrinsic parameters of the shooting device and the specific shooting parameters of the shooting device, a preset effective field of view area is determined, and the fisheye image is obtained based on the shooting device; If a pixel in the fisheye image that is outside the preset effective field of view is marked as being within the preset effective field of view, then the pixel in the fisheye image that is outside the preset effective field of view is marked as a background pixel.

2. The method according to claim 1, characterized in that, The preset model includes a convolutional layer; wherein the convolutional kernel used in the convolutional layer has holes set based on a preset dilation rate.

3. The method according to claim 2, characterized in that, The preset model also includes a fully connected layer and a spatial pyramid pooling layer; the spatial pyramid pooling layer is used to convert the feature vector output by the convolutional layer into a fixed-length feature vector and input it into the fully connected layer.

4. The method according to any one of claims 1 to 3, characterized in that, The preset model includes an attention layer; wherein, the feature corresponding to each pixel in the fisheye image is obtained by weighted aggregation of the features of pixels at other locations in the fisheye image through the attention layer.

5. The method according to any one of claims 1 to 3, characterized in that, The preset model is obtained by compression based on a preset method or is reconstructed based on a backbone network. The backbone network is any one of the following models: MobileNetv2, MnasNet, FCN, HRNet, FastSCNN, and PSPNet. The preset method includes any one of the following methods: weight reduction, weight decomposition, data precision reduction, and weight sharing.

6. The method according to any one of claims 1 to 3, characterized in that, The preset model also includes an upsampling layer, which is used to upsample the feature map generated after the fisheye image is input into the preset model.

7. The method according to any one of claims 1 to 3, characterized in that, After inputting the fisheye image into a preset model and outputting the labeling result of each pixel in the fisheye image, the method further includes: Based on conditional random fields, the labeling results of each pixel in the fisheye image are corrected.

8. A fisheye image semantic segmentation device, characterized in that, The device includes: The output module is used to input the fisheye image into the preset model and output the labeling result of each pixel in the fisheye image. The preset model is obtained after training based on the sample fisheye image, and each pixel in the sample fisheye image has been labeled. The determination module is used to determine a preset effective field of view area based on the intrinsic and extrinsic parameters of the shooting device and the specific shooting parameters of the shooting device, wherein the fisheye image is obtained based on the shooting device; The marking module is used to mark the pixels in the fisheye image that are outside the preset effective field of view as background pixels when the pixels in the fisheye image that are outside the preset effective field of view are marked as being within the preset effective field of view.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images