A multi-view joint semantic segmentation method for surround view fisheye images

Abstract

The application relates to the technical field of image semantic segmentation, and discloses a multi-view joint semantic segmentation method for surround-view fisheye images, a multi-view joint segmentation model and a training process, which comprises the following steps: preparing a training data set, taking fisheye images under multiple views at the same moment as a group of surround-view images, the training data set comprising multiple groups of surround-view images and corresponding pixel-level semantic annotations; establishing a multi-view joint segmentation model based on real-time semantic segmentation, which comprises a feature extraction module, a distortion detail supplement module, a multi-view context fusion module and a decoder module; and training the multi-view joint segmentation model based on the training data set and cross-entropy loss. According to the application, multi-view distortion completion is first performed on high-resolution detail features in joint segmentation, and then multi-view position-related context fusion is performed in a deep layer, so that the correlation information between multi-view fisheye images is utilized, and high-precision semantic segmentation of the fisheye images is realized.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation technology, and specifically to a multi-view joint semantic segmentation method for panoramic fisheye images. Background Technology

[0002] In recent years, image semantic segmentation, as one of the core tasks of computer vision, has demonstrated outstanding application value in many scenarios. In the field of autonomous driving, image semantic segmentation, along with inverse perspective transformation and camera calibration, enables pixel-level distance perception of objects. However, the widely studied pinhole image segmentation method has encountered new problems with fisheye images, which are extensively used in autonomous driving. Due to the severe radial distortion caused by fisheye cameras, the pixel distribution in fisheye images is extremely uneven. Objects in the center of the image occupy more pixel information, while objects at the edges have very low pixel information density and a large amount of distortion context information, which greatly affects the image segmentation ability of convolutional neural networks. Traditional methods for solving fisheye image segmentation involve designing special convolutional kernels to enable neural network models to fit individual distorted images, such as deformable convolutions and their various variants.

[0003] However, in the application of image semantic segmentation technology in the field of autonomous driving, images are often not fed into the model for parsing as a serial data stream. Instead, they are fed into a parallel data stream formed by combining multiple viewpoint images from the same time frame and processed in parallel by the GPU. This form of data stream provides a new solution for fisheye image segmentation. Because fisheye images have the characteristics and advantages of extremely high FOV (Field of View), and the joint field of view of the fisheye cameras mounted on vehicles far exceeds the 360° panoramic range (typically configured with 4-8 fisheye cameras), objects appearing at the edge of the image from viewpoint A often appear in the center and surrounding area of ​​the image from viewpoint B (adjacent). Therefore, although an object may have extremely low pixel density and highly distorted contextual information in the image from viewpoint A, it can benefit from the high-quality information in the central region of the fisheye image in viewpoint B, obtaining good detail and contextual features, and thus achieving sufficiently good segmentation accuracy in the image from viewpoint B. This invention aims to utilize this special correlation information between multiple fisheye viewpoint images to build a joint segmentation framework for fisheye images. Semantic segmentation methods integrating multiple image-related information include multi-head self-attention modeling of image patches under the Vision Tansfomer architecture, and feature interaction fusion methods between image patches in high-resolution image segmentation. In the patent document "A Cueing Fine-tuning Method Based on a Large Visual Model," multi-head self-attention first performs uniform multi-grid segmentation of the image, encodes each small image, and uses it as a token to model correlation relationships in a large number of stacked self-attention modules. Finally, it decodes the deep image tokens and outputs the corresponding semantic segmentation results. In the patent document "A Strong Fusion Remote Sensing Image Segmentation Method with High Resolution and Pixel Relationship Enhanced Attention," pixel relationship enhancement and channel attention modules are added when fusing features of local and global images. These modules then perform convolutional mapping on the local feature maps. However, the above methods either rely on a large number of global self-attention modules, resulting in significant unnecessary attention computation costs, or primarily explore the joint context of local and adjacent images. They do not model or utilize the joint relationships of diverse representations of the same object under different viewpoints, and fail to capture the detailed information missing from the edges of fisheye images.

[0004] In summary, most semantic segmentation methods for fisheye images use deformable convolution-guided models to model distortion characteristics, failing to leverage the unique high field of view advantage of fisheye cameras in panoramic segmentation to capture usable information between adjacent viewpoints. Furthermore, methods for jointly utilizing related images are mostly based on multi-head self-attention modules or naive feature fusion, lacking effective ways to utilize relevant information between multiple fisheye domain images in panoramic view. Therefore, this invention proposes a novel panoramic fisheye image segmentation method from the perspective of multi-view joint segmentation. This method utilizes the unique distortion correlation and contextual complementarity between multi-view fisheye images, integrating high-quality detail features and weak distortion contextual features from each viewpoint to address the problem of poor detail in fisheye image edge region segmentation, resulting in better segmentation results. Summary of the Invention

[0005] To address the issue of semantic segmentation accuracy in surround-view fisheye images in the field of autonomous driving, this invention discloses a multi-view joint semantic segmentation method for surround-view fisheye images. This method can supplement distortion details of the same object from different viewpoints and fuse diverse contexts, effectively enhancing the model performance of fisheye image semantic segmentation and enabling the modeling and utilization of correlation information between multi-view images.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A multi-view joint semantic segmentation method for panoramic fisheye images involves inputting fisheye images from multiple views simultaneously into a trained multi-view joint segmentation model to generate segmentation maps. The construction and training process of the multi-view joint segmentation model specifically includes the following steps:

[0008] Step 1: Prepare the training dataset. Take fisheye images from multiple perspectives at the same time as a set of panoramic images. The training dataset includes multiple sets of panoramic images and corresponding pixel-level semantic annotations.

[0009] Step 2: Establish a multi-view joint segmentation model based on real-time semantic segmentation, including a feature extraction module, a distortion detail completion module, a multi-view context fusion module, and a decoder module. The surrounding image input to the multi-view joint segmentation model is processed by the feature extraction module to obtain detail features and context features for each viewpoint. The distortion detail completion module models the radial distortion of the fisheye image at minimal cost, using distortion information from adjacent views to supplement detail features. An auxiliary segmentation mask is obtained through deformable convolution and an auxiliary segmentation head. The segmentation loss is calculated to regularize and constrain the distortion detail features, and the distortion detail features are then completed. The multi-view context fusion module evaluates the correlation by calculating the inner product of the context features of adjacent views and the current viewpoint, using the correlation matrix to focus on relevant semantic regions and enhance the context features. The completed detail features and enhanced context features are fused, and the resulting super multi-view features are input into the main segmentation head in the decoder module for decoding to output a segmentation map.

[0010] Step 3: Train the multi-view joint segmentation model based on the training dataset and cross-entropy loss; after the model training reaches equilibrium, the segmentation map generated by the decoder module is the segmentation result of the multi-view joint segmentation model on the input image.

[0011] Furthermore, the multi-view joint segmentation model adopts an encoder-decoder structure, where feature flows from different views interact to generate super multi-view features.

[0012] The feature extraction module is used as an encoder; the multi-view fisheye image I i The input feature extraction module performs feature extraction and outputs context features from different perspectives. and detailed features i represents the viewpoint index of the fisheye image, i = [1, ..., n], where n is the number of viewpoints in a set of panoramic images; the contextual features and detail features of the viewpoint to the left of the i-th viewpoint are respectively represented as... The contextual features and detail features of the viewpoint right adjacent to the i-th viewpoint are respectively represented as:

[0013] The decoder module includes the main segmentation header Dec. i and auxiliary segmentation head Segmentation P i The main segmentation head Dec of the multi-view joint segmentation model i Output:

[0014]

[0015] in, This indicates a super multi-view feature.

[0016] Furthermore, there are n decoder modules in total. Each decoder module and the same feature extraction module form an encoder-decoder structure, and there are a total of n encoder-decoder structures. Fisheye images from n perspectives in a set of panoramic images are input into the same feature extraction module for parallel processing. The output of the feature extraction module is split into multiple perspectives and input into the distortion detail supplementation module and the multi-view context fusion module for processing. The obtained super multi-view features are input into the decoder to obtain the segmentation map.

[0017] Furthermore, the distortion detail supplementation module is used to model the radial distortion of the fisheye image at minimal cost, and to supplement the detail features in the current viewpoint by utilizing the dilated distortion information of the same object in adjacent viewpoints; the detail features corresponding to each viewpoint are input into the deformable convolution Dconv, and the obtained distortion detail features are input into the auxiliary segmentation head to obtain the auxiliary segmentation mask. And based on auxiliary segmentation mask The segmentation loss is calculated using the real mask in the i-th viewpoint, which is used to regularize the distortion detail features; the distortion detail features in the i-th viewpoint are then completed pixel by pixel to obtain the completed distortion detail features.

[0018] Furthermore, the step of performing pixel-by-pixel completion on the distortion detail features of the i-th viewpoint to obtain the completed distortion detail features specifically includes:

[0019] For each feature vector p on the distortion detail features k The distortion detail features from adjacent viewpoints are matched for feature correlation to obtain a position weight matrix. Then, the distortion detail features from adjacent viewpoints are weighted and summed using the position weight matrix. After concatenation with the original distortion detail features in the channel dimension, norm regularization and convolution are performed to obtain the final complete distortion detail features.

[0020] Furthermore, the purpose of the multi-view context fusion module is to capture the missing context information in the current view from neighboring views; firstly, it evaluates the context features of the left neighboring view and the right neighboring view. Features of the main perspective context The correlation is calculated by the inner product between features; the correlation matrix is ​​used as an attention map to enhance the context features of each viewpoint, resulting in context features of each viewpoint enhanced by position correlation.

[0021] Weights are calculated for the context features of each viewpoint enhanced by positional relevance. A weighted sum of these enhanced context features is then calculated using these weights. This sum is merged with the completed distortion detail features output by the feature fusion module and the distortion detail enhancement module. Finally, the fused features from each viewpoint are channel-fused with the context features of the current viewpoint to obtain the super multi-view feature corresponding to the current viewpoint.

[0022] Furthermore, in step three, training the multi-view joint segmentation model based on the training dataset and cross-entropy loss specifically includes:

[0023] Constructing cross-entropy loss l ce :

[0024]

[0025] Among them, P i This represents the segmentation map output by the multi-view joint segmentation model for the fisheye image of the i-th view, where N is the batch size of the data group.

[0026] The overall loss L of the multi-view joint segmentation model constructed based on cross-entropy loss:

[0027]

[0028] Where W represents the model weights, α represents the weights of the auxiliary detail loss, and Dconv represents deformable convolution.

[0029] Compared with the prior art, the beneficial technical effects of the present invention are:

[0030] 1. This invention utilizes the information correlation between multi-view fisheye images captured by multiple fisheye cameras during vehicle movement and the inherent field-of-view distortion characteristics of fisheye cameras to effectively enhance the model's segmentation performance for compressed distortion objects on the periphery of a single-view distorted image.

[0031] 2. This invention proposes an interaction method for high-resolution detail features between multi-view images. It utilizes deformable convolution to refine distorted detail features and supplements and refines the high-resolution detail features of the main viewpoint through the positional relationship weight matrix of features between multiple views.

[0032] 3. This invention proposes a method for utilizing multi-view joint context to complete high-dimensional completion of the context of a single viewpoint at the 2D level.

[0033] In summary, this invention first performs multi-view distortion completion on the high-resolution detail features in joint segmentation, and then fuses multi-view position-related contexts at a deeper level of the model. By utilizing the correlation information between multi-view fisheye images, a novel multi-view joint segmentation model is built to achieve high-precision semantic segmentation of fisheye images. Attached Figure Description

[0034] Figure 1 Explanation of the complementarity of panoramic fisheye images;

[0035] Figure 2 This is a schematic diagram of the multi-view joint segmentation model in this invention;

[0036] Figure 3 This is a schematic diagram of the Distortion Detail Compensation Module (DDRM) between adjacent viewpoints in this invention;

[0037] Figure 4 This is a schematic diagram of the multi-view context fusion module (LCFM) between adjacent viewpoints in this invention. Detailed Implementation

[0038] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0039] In the field of autonomous driving, perception is the foundation of all algorithmic tasks, and semantic segmentation can bring pixel-level perception accuracy. Multi-view fisheye camera systems are widely used in numerous scenarios due to their ultra-large field of view and blind-spot-free characteristics. However, traditional semantic segmentation methods have low performance on fisheye images, especially given the edge compression distortion of fisheye images, which poses a challenge to semantic segmentation. First, this invention innovatively designs a segmentation framework for surround-view fisheye camera systems. Utilizing the complementary information characteristics between multiple simultaneous fisheye images from different perspectives, a novel multi-view joint segmentation model is built, providing a new optimization perspective for image semantic segmentation technology. Second, this invention proposes a distortion detail supplementation module and a position-related multi-view context fusion module, establishing a new information interaction mode between related images, thereby improving the model's semantic segmentation performance on fisheye images.

[0040] This invention discloses a multi-view joint semantic segmentation method for panoramic fisheye images. The proposed multi-view joint segmentation model comprises three core parts: the overall design of the joint segmentation framework (e.g., Figure 2 As shown), distortion detail supplementation module (such as...) Figure 3 As shown), a location-related multi-view context fusion module (such as...) Figure 4(As shown). The overall design of the joint segmentation framework is based on a single-view real-time semantic segmentation model. Two interactive modules focus on optimizing the most critical detail features and contextual features in the semantic segmentation task, respectively, and interact with features from adjacent views based on these. First, in the joint segmentation framework, multiple fisheye images from different views are input simultaneously. These images are uniformly processed by a feature extraction module to obtain detail features and contextual features for each view. Then, adjacent views are augmented using a distortion detail module to obtain high-level detail features, which are simultaneously constrained and optimized by an auxiliary segmentation head. Next, a multi-view context fusion module selectively extracts and fuses relevant context from adjacent views. Finally, the resulting super multi-view features are decoded by the main segmentation head to output a segmentation map. After the multi-view joint segmentation model reaches equilibrium during training, the segmentation map generated by the decoder is the framework's prediction result for the input image.

[0041] The joint segmentation framework employs an encoder-decoder structure, where feature flows from different viewpoints interact to generate high-level multi-view features. To extract low-level high-resolution detail features and deep low-resolution contextual features from different viewpoints, fisheye images I of size H×W from each viewpoint are first processed... i A unified encoder Enc is input for feature extraction, where i = [1,…n], and n is the number of views. This encoder outputs high-level contextual features. Its feature resolution is 1 / 64 of the input resolution size, and it can extract shallow detail features from the feature stream. Its resolution is 1 / 8 of the input resolution. The encoder employs a feature extraction module. The detail features and contextual features of adjacent (left-adjacent -, right-adjacent +) viewpoints are represented as follows: These features will be used for multi-view feature interaction as described below. For a particular viewpoint's decoder module, in addition to including the main segmentation head Dec for that viewpoint... i In addition, there is an auxiliary segmentation head. Used to constrain distortion detail features. The final segmentation map is output by the main decoder of the entire segmentation framework. in The super multi-view features, obtained through the interaction of multi-view feature streams, will be described later. The difference between this framework and traditional semantic segmentation models lies in the feature splitting and interaction within the same data set. For example, a traditional segmentation model takes four multi-view images as a batch input to the encoder-decoder (a feature stream), while this framework processes the feature streams of the four images separately. This can be understood as using four encoder-decoders for processing and synchronous communication between the models. However, the four models share the same feature extraction module and can process the feature extraction stages of all images in parallel with a batch size of four, thus minimizing the increase in overall model computational overhead. After feature calculation at each resolution, multi-view splitting is performed, and the results are input into the distortion detail supplementation module and the multi-view context-selective fusion module for feature processing. Finally, the obtained super multi-view features are input into the decoder to obtain a high-precision segmentation map.

[0042] The purpose of the distortion detail enhancement module is to model the radial distortion of fisheye images at minimal cost and supplement low-resolution detail features in the main viewpoint by utilizing the dilated distortion information of the same object in adjacent viewpoints. For example... Figure 1 As shown, in fisheye images, it is evident that due to their radial distortion characteristics, objects at the image edges are significantly compressed, resulting in a substantial loss of detail. This is devastating for pixel-intensive tasks such as semantic segmentation, leading to low segmentation accuracy. However, in a panoramic fisheye camera system, objects appearing at the edge of an image from a certain viewpoint are more likely to appear in the center or near-center region of adjacent viewpoints, possessing higher pixel density and detail information. Therefore, this invention designs a distortion detail supplementation module between multiple viewpoints. First, the detailed features obtained from each viewpoint are input into the deformable convolution Dconv. Since the number of parameters and computational cost of deformable convolution is often several times that of ordinary convolution, we do not use it in the main feature stream, but rather in the distortion detail supplementation module to model the distortion of the fisheye image at minimal cost. Then, the high-resolution distortion detail features are input into the auxiliary segmentation head to obtain the auxiliary segmentation mask. The segmentation loss is then calculated using this loss and the true mask for that viewpoint, and used to regularize the distortion detail features. Finally, pixel-by-pixel completion is performed on the distortion detail features for that viewpoint. Specifically, for each feature vector p on the distortion detail features... k Distortion detail features compared to adjacent viewpoints (taking the left adjacent viewpoint as an example). A location weight matrix is ​​obtained by performing feature correlation matching.

[0043]

[0044] Then, the position weight matrix is ​​input into the Softmax function to perform a weighted sum of the distortion detail features from adjacent viewpoints. After concatenation with the original features in the channel dimension, the sum is subjected to norm regularization and a 1×1 convolution to obtain the final complete distortion detail features. (The calculations related to the right-side view are the same). For ease of representation, this module can be uniformly written as...

[0045] The purpose of the location-related multi-view context fusion module is to capture contextual information missing in the current view from adjacent viewpoints. In fisheye images, all objects and scene sizes exhibit radial distortion. Compression distortion in edge regions and gradual distortion along the distortion radius extension line all affect the contextual features obtained from the fisheye image and negatively impact semantic segmentation. Therefore, in a surround-view camera system, the design objective of this invention is to obtain diverse contexts of the same object from adjacent viewpoints, as well as complete contexts of objects that are only partially captured. First, in the location-related context correction section, the contextual features of each adjacent viewpoint are evaluated. Features of the main perspective context The correlation is calculated using the inner product of features (taking the left neighbor perspective as an example). By establishing pairwise pixel-level relationships to measure non-local relevance, this relevance matrix can be further used as an attention map to enhance the contextual features of each viewpoint, making it more focused on semantic regions related to the main viewpoint's context. Specifically, by obtaining the attention map through a softmax layer and then performing an inner product with each viewpoint, the multi-view context corrected for positional relevance can be obtained. and C i+ (The calculation principle is the same as above). Then, in the multi-view context fusion part, the weight map {H,H} is calculated for the context features of each view based on the features obtained above. - The computation process is accomplished through a squeezing and splitting module, which includes a 1×1 convolutional layer that compresses the connected features, another 1×1 convolutional layer that reconstructs the compressed features back to the original dimensions, and finally a softmax layer to obtain the normalized weights for each viewpoint. These weights are then used to calculate the weighted sum of the contextual features relevant to each location, and finally merged with the distortion-supplemented detail features through a residual-designed feature fusion module.

[0046]

[0047] For clarity and conciseness, the following description will be used. Similarly, we can obtain Will With the main perspective context Input a 1×1 convolution to perform channel fusion and obtain super multi-view features.

[0048]

[0049] The super multi-view features are input into the master segmentation head of this view. i Decode the image to obtain the predicted segmentation map.

[0050] The overall training loss function of the multi-view joint segmentation model in this invention uses cross-entropy loss:

[0051]

[0052] Among them, P i This represents the segmentation map output by the multi-view joint segmentation model for the fisheye image of the i-th view, where N is the batch size of the data group.

[0053] The overall loss L of the multi-view joint segmentation model includes the main loss of each view and the auxiliary loss of the auxiliary branches:

[0054]

[0055] Where W represents the weights of the multi-view joint segmentation model, and n represents the number of views.

[0056] Example

[0057] This invention provides a joint segmentation method for panoramic fisheye images based on information interaction between multiple viewpoints. Since it is designed for real-time segmentation scenarios, and considering the lightweight nature and mature engineering optimization of Convolutional Neural Network (CNN) models, the joint segmentation framework uses a CNN-based real-time segmentation model as its foundation. First, multiple distorted images captured by fisheye cameras at the same time are obtained. Then, all images in the same frame are fed into the multi-view joint segmentation model, which predicts the semantic category of each pixel in each image.

[0058] Step 1: Prepare the training dataset. This requires all images from multiple viewpoints at the same time point to be grouped together, and the dataset should contain multiple sets of panoramic images. Each image is scaled to maintain a uniform size, normalized, and then preprocessed using standard image enhancement techniques (random cropping is not allowed). The final data includes all groups of panoramic images and their corresponding pixel-level semantic annotations.

[0059] Step 2: Using a deep learning framework, establish a multi-view joint segmentation model based on real-time semantic segmentation (CNN series), consisting of a feature extraction module, a distortion detail enhancement module, a multi-view context fusion module, and a decoder module. The feature extraction module can be a mainstream real-time semantic segmentation model, such as BiseNet. In this embodiment, RegSeg is used as the feature extraction module and its corresponding decoder. The feature extraction modules for each viewpoint use the same model, but each uses its own decoder module and distortion detail enhancement module.

[0060] Step 3: Based on the composition of the dataset, construct a surround-view fisheye camera system with four perspectives: front, back, left, and right. That is, fisheye images from the four perspectives at the same time are considered as a set of data. It can be clearly observed that the left and right adjacent perspectives of a given view contain a large amount of relevant information, while the opposing perspective contains almost none. Therefore, the surround-view joint segmentation framework of this invention only performs information interaction between adjacent perspectives. For example, for the feature flow of the front-view fisheye image, only the feature flows of the left and right perspective images are considered to interact and supplement it; the same applies to the other three perspectives.

[0061] Step 4: Input multiple sets of panoramic images into the feature extraction module to obtain high-resolution detail features (1 / 8 initial resolution) and deep high-level context features (1 / 64 initial resolution) that include all perspectives (front, back, left, right). At this point, the perspectives can be treated as a unified batch without distinguishing between them, making full use of GPU parallel computing resources.

[0062] Step 5: Distinguish the image feature streams from each viewpoint direction and exchange information with the features from adjacent viewpoints. The following explanation uses the forward viewpoint as an example; the same applies to the other three viewpoints.

[0063] Step 6: The detail features of the forward-facing image, along with the detail features of the left and right-facing images, are input into the distortion detail supplementation module. First, each detail feature is passed through a deformable convolutional layer to obtain distortion detail features. Here, each distortion detail feature is paired with a decoder for segmentation loss calculation. Furthermore, the forward-facing distortion detail features are matched with the distortion detail features of the left and right-facing images to obtain corresponding positional weights. Then, pixel-level detail feature weighted summation is performed. Finally, channel-level feature merging and channel integration using a 1×1 convolutional layer are performed to output the supplemented distortion detail features for the forward-facing image.

[0064] Step 7: The contextual detail features from the forward view, along with the left and right contextual features, are input into the multi-view context fusion module. First, the contextual features from the three views are corrected for positional relevance based on the forward contextual features, and then context fusion is performed based on the calculated weight matrix. Finally, the distortion detail features output from the distortion supplementation module are input into the residual feature fusion module to obtain the final super multi-view features, and these features are used to input the decoder to calculate the segmentation loss.

[0065] Step 8: Repeat steps 6 and 7 to process the relevant feature flow and calculate the relevant loss function for the remaining three perspectives.

[0066] Step 9: Based on the loss functions obtained in the previous steps, minimize the loss functions using the backpropagation algorithm and gradient descent strategy, and update the weight parameters of each module of the model.

[0067] Step 10: Input the test dataset and calculate the accuracy of the segmentation results output by the model.

[0068] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0069] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-view joint semantic segmentation method for panoramic fisheye images, characterized in that, Simultaneously, fisheye images from multiple viewpoints are input into a trained multi-view joint segmentation model to generate segmentation maps. The construction and training process of the multi-view joint segmentation model includes the following steps: Step 1: Prepare the training dataset. Take fisheye images from multiple perspectives at the same time as a set of panoramic images. The training dataset includes multiple sets of panoramic images and corresponding pixel-level semantic annotations. Step two: Establish a multi-view joint segmentation model based on real-time semantic segmentation, including a feature extraction module, a distortion detail enhancement module, a multi-view context fusion module, and a decoder module; the surrounding image input to the multi-view joint segmentation model, after passing through the feature extraction module, obtains the detail features and context features of each view: the multi-view fisheye image The input feature extraction module performs feature extraction and outputs context features from different perspectives. and detailed features , Represents the viewpoint index of a fisheye image. , Let be the number of viewpoints in a set of panoramic images; the contextual features and detail features of the viewpoint to the left of the i-th viewpoint are respectively represented as: The context features and detail features of the viewpoint right adjacent to the i-th viewpoint are respectively represented as: ; The distortion detail enhancement module models radial distortion in fisheye images at minimal cost, supplements detail features using distortion information from adjacent viewpoints, obtains an auxiliary segmentation mask through deformable convolution and an auxiliary segmentation head, calculates the segmentation loss to regularize and constrain the distortion detail features, and completes the distortion detail features. The multi-view context fusion module evaluates the correlation by calculating the inner product of the context features of adjacent viewpoints and the current viewpoint, uses the correlation matrix to focus on relevant semantic regions, and enhances the context features. The completed detail features and the enhanced context features are fused, and the resulting super multi-view features are input into the main segmentation head in the decoder module for decoding to output a segmentation map. Step 3: Train the multi-view joint segmentation model based on the training dataset and cross-entropy loss; after the model training reaches equilibrium, the segmentation map generated by the decoder module is the segmentation result of the multi-view joint segmentation model on the input image. The purpose of the multi-view context fusion module is to capture the missing context information in the current view from neighboring views; firstly, it evaluates the context features of the left and right neighboring views. Features of the main perspective context The correlation is calculated by the inner product between features; the correlation matrix is ​​used as an attention map to enhance the context features of each viewpoint, and the context features of each viewpoint are corrected for position correlation. Weights are calculated for the context features of each viewpoint after position correlation correction. A weighted sum of the position correlation-corrected context features of each viewpoint is then calculated using these weights, and finally merged with the completed distortion detail features output by the distortion detail supplementation module through the feature fusion module. ； The context features of the left-hand perspective have been corrected for positional relevance. For weighted graphs; Similarly, we can obtain ; To complete the distortion details from the current perspective; Will , Features of the main perspective context Input a 1×1 convolution to perform channel fusion and obtain super multi-view features. ; The distortion detail enhancement module inputs the detail features corresponding to each viewpoint into a deformable convolution. In the process, the obtained distortion detail features are input into the auxiliary segmentation head of the decoder of the multi-view joint segmentation model. To obtain the auxiliary segmentation mask ; and based on auxiliary segmentation masks And the segmentation loss is calculated using the real mask in the i-th view, which is used to regularize the distortion detail features; The distortion detail features of the i-th viewpoint are completed pixel-by-pixel to obtain the completed distortion detail features. Specifically, this includes: for each feature vector of the distortion detail features... The system performs feature correlation matching with the distortion detail features of adjacent viewpoints to obtain a position weight matrix. Then, the position weight matrix is ​​used to perform weighted summation of the distortion detail features of adjacent viewpoints. After concatenation with the distortion detail features of the current viewpoint in the channel dimension, norm regularization and convolution are performed to obtain the final complete distortion detail features.

2. The multi-view joint semantic segmentation method for panoramic fisheye images according to claim 1, characterized in that, The multi-view joint segmentation model adopts an encoder-decoder structure, and the feature streams between different views interact to generate super multi-view features. The feature extraction module is used as an encoder; The decoder module includes the main segmentation head. and auxiliary segmentation head ; Segmentation image The main segmentation head of the multi-view joint segmentation model Output: ； in, This indicates a super multi-view feature.

3. The multi-view joint semantic segmentation method for panoramic fisheye images according to claim 2, characterized in that, There are n decoder modules in total. Each decoder module and the same feature extraction module form an encoder-decoder structure, and there are a total of n encoder-decoder structures. Fisheye images from n perspectives in a set of panoramic images are input into the same feature extraction module for parallel processing. The output of the feature extraction module is split into multiple perspectives and input into the distortion detail supplementation module and the multi-view context fusion module for processing. The resulting super multi-view features are input into the decoder to obtain the segmentation map.

Images