Multi-view joint semantic segmentation method for look-around fisheye image

Abstract

The invention relates to the technical field of image semantic segmentation, and discloses a multi-view joint semantic segmentation method for looking-around fisheye images, and a multi-view joint segmentation model construction and training process, and the method comprises the steps: preparing a training data set, taking the fisheye images at the same moment under a plurality of views as a group of looking-around images, and carrying out the training of the group of looking-around images; the training data set comprises multiple groups of look-around images and corresponding pixel-level semantic annotations; establishing a multi-view joint segmentation model based on real-time semantic segmentation, wherein the multi-view joint segmentation model comprises a feature extraction module, a distortion detail supplement module, a multi-view context fusion module and a decoder module; and training the multi-view joint segmentation model based on the training data set and the cross entropy loss. According to the method, firstly, multi-view distortion completion is carried out on high-resolution detail features in joint segmentation, and then fusion of multi-view position related contexts is deeply carried out, so that high-precision semantic segmentation of the fisheye images is realized by utilizing associated information among the multi-view fisheye images.

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to a multi-view joint semantic segmentation method for surround fisheye images. Background Art

[0002] In recent years, image semantic segmentation, as one of the core tasks of computer vision, has shown excellent application value in many scenarios. Among them, in the field of autonomous driving, pixel-level distance perception of objects is obtained through image semantic segmentation, inverse perspective transformation and camera calibration. However, the widely studied pinhole image segmentation method has encountered new problems under the fisheye images that are widely used in the field of autonomous driving. Due to the severe radial distortion caused by the fisheye camera, the pixel distribution in the fisheye image is extremely uneven. The objects in the middle of the image have more pixel information, while the objects at the edge of the image have extremely small pixel information density and extremely large distortion context information, which has a great impact on the image segmentation ability of the convolutional neural network. The traditional method to solve the problem of fisheye image segmentation is to design a special convolution kernel so that the neural network model can obtain the ability to fit a single distorted image, such as deformable convolution and its various variants.

[0003] However, when applying image semantic segmentation technology in the field of autonomous driving, images are often not sent to the model for analysis as a serial data stream, but a parallel data stream formed by combining multiple perspective images under the same time frame and sent to the GPU device for parallel processing. This form of data stream brings new solutions to the problem of fisheye image segmentation. Since the characteristics and advantages of fisheye images are that they have an extremely high FOV field of view, and the joint field of view of the fisheye camera mounted on the vehicle far exceeds the panoramic range of 360° (usually 4-8 fisheye cameras are configured), objects that appear at the edge of the A perspective image often appear in the center of the B (adjacent) perspective image and its surrounding area. Therefore, although an object has extremely low pixel density and extremely distorted context information in the A perspective image, it can benefit from the high-quality information brought by the central area of ​​the fisheye image in the B perspective, obtain good detail features and context features, and thus achieve sufficiently excellent segmentation accuracy in the B perspective image. The present invention aims to use the correlation information between this special fisheye multi-perspective image to build a joint segmentation framework for fisheye images. Semantic segmentation methods that integrate information related to multiple images include multi-head self-attention modeling of image blocks under the Vision Tansfomer architecture, and feature interaction fusion methods between image blocks in high-resolution image segmentation. In the patent document "A prompt fine-tuning method based on a large visual model", multi-head self-attention first performs uniform multi-grid segmentation on the image, encodes each small image and puts it into a large number of stacked self-attention modules as tokens for correlation relationship modeling, and finally decodes the deep image token and outputs the corresponding semantic segmentation result. In the patent document "High-resolution and pixel relationship enhanced attention strong fusion remote sensing image segmentation method", pixel relationship enhancement and channel attention modules are added when the local image and the global image are feature fused, and the pixel relationship enhancement and channel attention modules are used to perform convolution mapping on the local feature map. However, the above methods either rely on a large number of global self-attention modules, which brings a lot of unnecessary attention calculation costs, or mainly explore the joint context of the local image and the adjacent image. There is no modeling and utilization of the joint relationship of the diverse representations of the same object under different perspectives, and it is impossible to capture the detail information missing from the edge of the fisheye image.

[0004] In summary, the semantic segmentation of fisheye images mostly uses deformable convolution-guided models to model distortion characteristics, and does not take advantage of the high field of view unique to fisheye cameras to capture available information between adjacent view images during surround segmentation. Most methods for jointly utilizing related images are based on multi-head self-attention modules or naive feature fusion, and there is no effective way to utilize the relevant information between multiple surround fisheye domain images. Therefore, from the perspective of multi-view joint segmentation, the present invention proposes a new surround fisheye image segmentation method, which utilizes the unique distortion correlation and context complementarity between multi-view fisheye images, integrates high-quality detail features and weakly distorted context features from each view, solves the problem of poor segmentation details in the edge area of ​​fisheye images, and outputs better segmentation results. Summary of the invention

[0005] In order to solve the problem of semantic segmentation accuracy of surround fisheye images in the field of autonomous driving, the present invention discloses a multi-view joint semantic segmentation method for surround fisheye images, which can supplement the distortion details and fusion of diversified contexts of the same object under different viewpoints, effectively enhance the model performance of fisheye image semantic segmentation, and realize modeling and utilization of correlation information between multi-view images.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A multi-view joint semantic segmentation method for surround fisheye images, which inputs fisheye images of multiple views at the same time into a trained multi-view joint segmentation model to generate a segmentation map; the construction and training process of the multi-view joint segmentation model specifically includes the following steps:

[0008] Step 1: prepare a training data set, and use the fisheye images from multiple perspectives at the same time as a group of surround view images. The training data set includes multiple groups of surround view images and corresponding pixel-level semantic annotations;

[0009] Step 2, establish a multi-view joint segmentation model based on real-time semantic segmentation, including a feature extraction module, a distortion detail supplementation module, a multi-view context fusion module and a decoder module; the surround image input to the multi-view joint segmentation model, after passing through the feature extraction module, obtains the detail features and context features of each view; the distortion detail supplementation module models the radial distortion of the fisheye image at the minimum cost, uses the distortion information of adjacent viewpoints to supplement the detail features, obtains the auxiliary segmentation mask through deformable convolution and auxiliary segmentation head, calculates the segmentation loss to regularize the distorted detail features, and completes the distorted detail features; the multi-view context fusion module evaluates the correlation by calculating the inner product of the context features of the adjacent viewpoints and the current viewpoint, uses the correlation matrix to focus on the relevant semantic areas, and enhances the context features; the completed detail features and the enhanced context features are fused, and the obtained super multi-view features are input into the main segmentation head in the decoder module for decoding to output the segmentation map;

[0010] Step three, train the multi-view joint segmentation model based on the training data set and cross entropy loss; after the model training reaches equilibrium, the segmentation map generated by the decoder module is the segmentation result of the multi-view joint segmentation model for the input image.

[0011] Furthermore, the multi-view joint segmentation model adopts an encoder-decoder structure, and feature flows between different views interact to generate super multi-view features;

[0012] The feature extraction module is used as an encoder; the multi-view fisheye image I i Input feature extraction module to extract features and output context features from different perspectives and detail features i represents the view index of the fisheye image, i = [1, ..., n], n is the number of views in a set of surround images; the contextual features and detail features of the view adjacent to the left of the i-th view are expressed as The context features and detail features of the view right adjacent to the i-th view are expressed as

[0013] The decoder module includes the main segmentation header Dec i and auxiliary segmentation head Segmentation Map i The main segmentation head Dec of the multi-view joint segmentation model i Output:

[0014]

[0015] in, Represents super multi-view features.

[0016] Furthermore, there are n decoder modules in total, each decoder module and the same feature extraction module constitute an encoder-decoder structure, and there are n encoder-decoder structures in total; fisheye images under n perspectives in a set of surround images are input into the same feature extraction module for parallel processing, the output of the feature extraction module is split into multiple perspectives and input into a distortion detail supplement module and a multi-perspective context fusion module for processing, and the obtained super multi-perspective features are input into the decoder to obtain a segmentation map.

[0017] Furthermore, the distortion detail supplementation module is used to model the radial distortion of the fisheye image at a minimum cost, and use the dilation distortion information of the same object in the adjacent view to supplement the detail features in the current view; the detail features corresponding to each view are input into the deformable convolution Dconv, and the obtained distortion detail features are input into the auxiliary segmentation head to obtain the auxiliary segmentation mask And based on the auxiliary segmentation mask The segmentation loss is calculated based on the true mask at the i-th perspective to regularize and constrain the distorted detail features; the distorted detail features of the i-th perspective are completed pixel by pixel to obtain the completed distorted detail features.

[0018] Furthermore, the pixel-by-pixel completion of the distortion detail features of the i-th viewing angle to obtain the completed distortion detail features specifically includes:

[0019] For each eigenvector p on the distorted detail feature k , and perform feature correlation matching with the distortion detail features of adjacent perspectives to obtain a position weight matrix; then use the position weight matrix to perform weighted summation on the distortion detail features of adjacent perspectives, and after splicing with the original distortion detail features in the channel dimension, perform norm regularization and convolution to obtain the final completed distortion detail features.

[0020] Furthermore, the purpose of the multi-view context fusion module is to capture the contextual information missing from the current view in the adjacent views. First, the contextual features of the left adjacent view and the right adjacent view are evaluated. Main perspective context features The correlation is calculated by the inner product between features. The correlation matrix is ​​used as an attention map to enhance the context features of each perspective, and the context features of each perspective enhanced by position correlation are obtained.

[0021] The weights of the context features enhanced by position correlation of each view are calculated, and the weighted sum of the context features enhanced by position correlation of each view is calculated with the weights, and the complemented distortion detail features output by the feature fusion module and the distortion detail supplement module are merged, and the fusion features of each view obtained are channel-fused with the context features of the current view to obtain the super multi-view features corresponding to the current view

[0022] Furthermore, in step three, the multi-view joint segmentation model is trained based on the training data set and the cross entropy loss, specifically including:

[0023] Construct cross entropy loss l ce :

[0024]

[0025] Among them, P i represents the segmentation map output by the multi-view joint segmentation model for the fisheye image of the i-th view, and N is the batch number of the data group;

[0026] The overall loss L of the multi-view joint segmentation model is constructed based on the cross entropy loss:

[0027]

[0028] Among them, W is the model weight, α represents the weight of auxiliary detail loss, and Dconv represents deformable convolution.

[0029] Compared with the prior art, the beneficial technical effects of the present invention are:

[0030] 1. The present invention utilizes the information correlation between multi-view fisheye images captured by multiple fisheye cameras during vehicle driving and the inherent field of view distortion characteristics of fisheye cameras, effectively enhancing the model's segmentation performance for compressively distorted objects in the periphery of a single-view distorted image.

[0031] 2. The present invention proposes an interactive mode of high-resolution detail features between multi-view images, uses deformable convolution to refine the distorted detail features, and supplements and refines the high-resolution detail features of the main view through the positional relationship weight matrix of the features between multiple views.

[0032] 3. The present invention proposes a method for utilizing multi-perspective joint context to complete high-dimensional completion of the context of a single perspective at the 2D level.

[0033] In summary, the present invention first performs multi-view distortion completion on the high-resolution detail features in the joint segmentation, and then fuses the multi-view position-related context in the deep layer of the model. In this way, the correlation information between multi-view fisheye images is utilized to build a new multi-view joint segmentation model to achieve high-precision semantic segmentation of fisheye images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a complementary illustration of the surround fisheye image;

[0035] Figure 2 is a schematic diagram of a multi-view joint segmentation model in the present invention;

[0036] Figure 3 Schematic diagram of a distortion detail supplement module (DDRM) between adjacent viewing angles in the present invention;

[0037] Figure 4 Schematic diagram of a multi-view context fusion module (LCFM) between adjacent views in the present invention. DETAILED DESCRIPTION

[0038] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0039] In the field of autonomous driving, perception tasks are the basis of all algorithm tasks, and semantic segmentation can bring pixel-level perception accuracy to perception. In a large number of scenarios, multi-view fisheye camera systems are widely used due to their ultra-large field of view and no dead angle characteristics, but traditional semantic segmentation methods have low segmentation performance for fisheye images, especially the edge compression distortion of fisheye images brings challenges to the semantic segmentation task. First of all, the present invention innovatively designs the segmentation framework for the surround-view fisheye camera system, and utilizes the information complementary characteristics between multiple-view fisheye images at the same time to build a new multi-view joint segmentation model, which brings new optimization perspectives to image semantic segmentation technology. Secondly, the present invention proposes a distortion detail supplement module and a position-related multi-view context fusion module, which establishes a new information interaction mode between related images, thereby improving the model's semantic segmentation performance for fisheye images.

[0040] The present invention discloses a multi-view joint semantic segmentation method for surround fisheye images. The core content of the proposed multi-view joint segmentation model includes three parts: the overall design of the joint segmentation framework (such as Figure 2 As shown in the figure), distortion detail supplement module (as ... Figure 3 As shown in the figure), the position-dependent multi-view context fusion module (as shown in the figure), Figure 4As shown). The overall design of the joint segmentation framework is based on a single-view real-time semantic segmentation model. The two interactive modules focus on the optimization methods of the most critical detail features and context features in the semantic segmentation task, and interact with the adjacent view features based on this. First, in the joint segmentation framework, fisheye images of multiple views are input at the same time, and the detail features and context features of each view are uniformly obtained through the feature extraction module; then, the adjacent views of each view are completed through the distortion detail module to obtain high-level detail features, and the auxiliary segmentation head constrains and optimizes them at the same time; then the multi-view context fusion module selectively extracts and fuses the relevant context of adjacent views, and finally the super multi-view features are decoded by the main segmentation head to output the segmentation map. After the multi-view joint segmentation model training reaches equilibrium, the segmentation map generated by the decoder is the prediction result of the input image by the framework.

[0041] The joint segmentation framework adopts an encoder-decoder structure, and the feature flows between different views interact to generate high-level multi-view features. In order to extract low-level high-resolution detail features and deep low-resolution context features from different views, we first transform the fisheye images I of each view with a size of H×W into i Input a unified encoder Enc for feature extraction, i = [1, ... n], n is the number of viewpoints, this encoder outputs high-level context features Its feature resolution is 1 / 64 of the input resolution, and shallow detail features can be extracted from the feature stream. Its resolution is 1 / 8 of the input resolution size. The encoder uses a feature extraction module. The detail features and context features of the view adjacent to this view (left adjacent -, right adjacent +) are respectively expressed as These features will be used for the multi-view feature interaction below. For a decoder module of a certain view, in addition to including the main segmentation header Dec of that view, i In addition, there is an auxiliary segmentation head Used to constrain the distortion detail features. The final segmentation map is output by the main decoder of the entire segmentation framework. in It is a super multi-view feature obtained by the interaction of multi-view feature streams, which will be described later. The difference between this framework and the traditional semantic segmentation model is that the same data group is split and interacted. For example, the input of the traditional segmentation model is 4 multi-view images as a batch input model encoder-decoder (a bundle of feature streams), while this framework processes the feature streams of 4 images separately, which can be understood as using 4 encoder-decoders for processing and synchronous communication between models, but the 4 models share the same feature extraction module and can process the feature extraction stage of all images in parallel in batches of 4, so the overall model operation overhead is minimally increased. After the calculation of each resolution feature is completed, multi-view splitting is performed and input into the distortion detail supplement module and the multi-view context selective fusion module for feature processing, and finally the obtained super multi-view feature is input into the decoder to obtain a high-precision segmentation map.

[0042] The purpose of the distortion detail supplement module is to model the radial distortion of the fisheye image at the minimum cost, and to use the dilation distortion information of the same object in the adjacent view to supplement the low-resolution detail features in the main view. Figure 1 As shown in the figure, in the fisheye image, it can be clearly observed that due to its radial distortion characteristics, objects at the edge of the image are significantly compressed, resulting in a large loss of details, which is devastating for pixel-intensive tasks such as semantic segmentation, and thus leads to lower segmentation accuracy. In the surround-view fisheye camera system, it can be found that objects appearing at the edge of an image at a certain perspective will appear at the center or more central area of ​​an adjacent perspective with a high probability, and have higher pixel density and detail information. Therefore, the present invention designs a distortion detail supplement module between multiple perspectives. First, the detail features obtained from each view are input into the deformable convolution Dconv. Since the number of parameters and the amount of calculation of the deformable convolution are often several times that of the ordinary convolution, we do not use it in the main feature flow, but instead use it in the distortion detail supplement module to model the distortion of the fisheye image at the minimum cost; then, the high-resolution distortion detail features are input into the auxiliary segmentation head to obtain the auxiliary segmentation mask. The segmentation loss is calculated with the real mask of the view to regularize the distortion detail features; finally, the distortion detail features of the view are completed pixel by pixel. Specifically, for each feature vector p on the distortion detail features k , and the distortion detail features of the adjacent view (taking the left adjacent view as an example) Perform feature correlation matching to obtain a position weight matrix

[0043]

[0044] Then the position weight matrix is ​​input into the Softmax function to make a weighted sum of the adjacent view distortion detail features. After splicing with the original features in the channel dimension, the final completed distortion detail features are obtained through norm regularization and 1×1 convolution. (The calculations for the right view are the same.) For ease of representation, this module can be written as

[0045] The purpose of the position-dependent multi-view context fusion module is to capture the contextual information missing from the current view in the adjacent view. In fisheye images, all objects and scenes will produce radial distortion. The compression distortion in the edge area and the gradual distortion on the extension line of the distortion radius will affect the contextual features obtained from the fisheye image and have a negative impact on semantic segmentation. Therefore, in the surround camera system, it is the original intention of the present invention to obtain the diverse context of the same object from adjacent viewpoints and the complete context of some objects that are only half captured. First, in the position-dependent context correction part, the contextual features of each adjacent viewpoint are evaluated. Main perspective context features The correlation is calculated by the inner product between features (taking the left adjacent view as an example) By establishing paired pixel-level relationships to measure non-local correlation, this correlation matrix can be further used as an attention map to enhance the contextual features of each view, making it more focused on the semantic areas related to the main view context. Specifically, the attention map is obtained through the Softmax layer and then the inner product is performed with each view to obtain the multi-view context corrected by position correlation. and C i+ (The calculation principle is the same as above.) Then, in the multi-view context fusion part, the weight map {H,H -}, the calculation process is completed through a squeeze and split module, which includes a 1×1 convolution layer to compress the connected features, another 1×1 convolution layer to reconstruct the compressed features to the original dimension, and finally obtain the normalized weights of each view through the softmax layer. The weighted sum of the relevant context features of each position is calculated with this weight, and finally merged with the distortion supplementary detail features through the residual design feature fusion module.

[0046]

[0047] For the sake of clarity and simplicity, the following description is used The same can be said Will With the main perspective context Input 1×1 convolution for channel fusion to obtain super multi-view features

[0048]

[0049] The super multi-view features are input into the main segmentation head Dec of this view. i Decode and get the predicted segmentation map

[0050] The overall training loss function of the multi-view joint segmentation model of the present invention uses cross entropy loss:

[0051]

[0052] Among them, P i Represents the segmentation map output by the multi-view joint segmentation model for the fisheye image of the i-th view, and N is the batch number of the data group.

[0053] The overall loss L of the multi-view joint segmentation model includes the main loss of each view and the auxiliary loss of the auxiliary branch:

[0054]

[0055] Among them, W is the weight of the multi-view joint segmentation model, and n is the number of views.

[0056] Example

[0057] The present invention provides a method for joint segmentation of surround fisheye images based on information interaction between multiple perspectives. Aiming at real-time segmentation scenarios, the joint segmentation framework uses a real-time segmentation model based on CNN as the basis, taking into account the lightness and maturity of engineering optimization of the convolutional neural network (CNN) series model. First, distorted images captured by multiple fisheye cameras at the same time are obtained; then all images in the same frame are sent to the multi-perspective joint segmentation model, and the corresponding semantic category of each pixel in each image is predicted by the multi-perspective joint segmentation model.

[0058] Step 1: Prepare a training dataset. All images at the same time under multiple perspectives are required to be a group. The dataset is required to contain multiple groups of surround view image data. Each image is scaled to keep a uniform size, normalized, and then preprocessed using some conventional image enhancement methods (random cropping cannot be used). The final data includes all the grouped surround view images and their corresponding pixel-level semantic annotations.

[0059] Step 2, using a deep learning framework, establish a multi-view joint segmentation model based on real-time semantic segmentation (CNN series), which consists of a feature extraction module, a distortion detail supplementation module, a multi-view context fusion module, and a decoder module. The feature extraction module can be the current mainstream real-time semantic segmentation model, such as BiseNet, etc. This embodiment uses RegSeg as the feature extraction module and its corresponding decoder. The feature extraction modules of each view use the same model, but use their own decoder modules and distortion detail supplementation modules.

[0060] Step 3, according to the composition of the data set, build a surround-view fisheye camera system with four perspectives: front, back, left, and right, that is, the fisheye images of the four perspectives at the same time are taken as a set of data. It can be clearly observed that the left and right adjacent perspectives of a perspective contain a lot of relevant information, while the opposite perspective almost does not exist. Therefore, in the surround-view joint segmentation framework of the present invention, only the information interaction between adjacent perspectives is performed. For example, for the feature stream of the front-view fisheye image, only the feature streams of the left and right perspective images are considered to interact and complement with it, and the same is true for the other three perspectives.

[0061] Step 4: Input multiple groups of surround view images into the feature extraction module to obtain high-resolution detail features (1 / 8 initial resolution) and deep high-level context features (1 / 64 initial resolution) covering all perspectives. At this time, no distinction is made between the perspectives and they can be treated as a unified batch to make full use of GPU parallel computing resources.

[0062] Step 5: distinguish the image feature streams in each viewing direction and exchange information with the features of the adjacent viewing angles. The following is an example of the forward viewing angle, and the same is true for the other three viewing angles.

[0063] In step 6, the detail features of the forward view image are input into the distortion detail supplementation module together with the detail features of the left and right views. First, each of them passes through a deformable convolution layer to obtain the distortion detail features. Here, the distortion detail features are first paired with a decoder to calculate the segmentation loss. In addition, the distortion detail features of the forward view are matched with the distortion detail features of the left and right views for feature similarity to obtain the corresponding position weights, and then the pixel-level detail features are weighted summed. Finally, the channel-level features are merged and the channels of the 1×1 convolution layer are integrated to output the supplementary distortion detail features of the forward view.

[0064] Step 7: The context detail features of the forward view are input into the multi-view context fusion module together with the left and right context features. First, the context features of the three views are position-correlatedly corrected according to the forward context features, and then the context fusion is completed according to the calculated weight matrix. Finally, the distortion detail features output by the distortion supplement module are input into the residual feature fusion module to obtain the final super multi-view features, and this feature is used to input the decoder to calculate the segmentation loss.

[0065] Step 8: Repeat steps 6 and 7 to process the relevant feature flows and calculate the relevant loss functions for the remaining three perspectives.

[0066] Step 9: According to the loss functions in the above steps, the back propagation algorithm and gradient descent strategy are used to minimize the loss function and update the weight parameters of each module of the model.

[0067] Step 10: Input the test data set and calculate the accuracy of the segmentation results output by the model.

[0068] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0069] In addition, it should be understood that although the present specification is described in terms of implementation modes, not every implementation mode contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A multi-view joint semantic segmentation method for surround fisheye images, characterized in that: The fisheye images of multiple views at the same time are input into the trained multi-view joint segmentation model to generate a segmentation map; the construction and training process of the multi-view joint segmentation model specifically includes the following steps: Step 1: prepare a training data set, and use the fisheye images from multiple perspectives at the same time as a group of surround view images. The training data set includes multiple groups of surround view images and corresponding pixel-level semantic annotations; Step 2, establish a multi-view joint segmentation model based on real-time semantic segmentation, including a feature extraction module, a distortion detail supplementation module, a multi-view context fusion module and a decoder module; the surround image input to the multi-view joint segmentation model, after passing through the feature extraction module, obtains the detail features and context features of each view; the distortion detail supplementation module models the radial distortion of the fisheye image at the minimum cost, uses the distortion information of adjacent viewpoints to supplement the detail features, obtains the auxiliary segmentation mask through deformable convolution and auxiliary segmentation head, calculates the segmentation loss to regularize the constraint of the distortion detail features, and completes the distortion detail features; the multi-view context fusion module evaluates the correlation by calculating the inner product of the context features of the adjacent viewpoints and the current viewpoint, uses the correlation matrix to focus on the relevant semantic areas, and enhances the context features; the completed detail features and the enhanced context features are fused, and the obtained super multi-view features are input into the main segmentation head in the decoder module for decoding to output the segmentation map; Step three, train the multi-view joint segmentation model based on the training data set and cross entropy loss; after the model training reaches equilibrium, the segmentation map generated by the decoder module is the segmentation result of the multi-view joint segmentation model for the input image.

2. The multi-view joint semantic segmentation method for surround fisheye images according to claim 1, characterized in that: The multi-view joint segmentation model adopts an encoder-decoder structure, and the feature streams between different views interact to generate super multi-view features; Using the feature extraction module as an encoder; The multi-view fisheye image I i Input feature extraction module to extract features and output context features from different perspectives and detail features i represents the view index of the fisheye image, i = [1, ..., n], n is the number of views in a set of surround images; the contextual features and detail features of the view adjacent to the left of the i-th view are expressed as The context features and detail features of the view right adjacent to the i-th view are expressed as The decoder module includes the main segmentation header Dec i and auxiliary segmentation head Segmentation Map i The main segmentation head Dec of the multi-view joint segmentation model i Output: in, Represents super multi-view features.

3. The multi-view joint semantic segmentation method for surround fisheye images according to claim 2, characterized in that: There are n decoder modules in total, each decoder module and the same feature extraction module form an encoder-decoder structure, and there are n encoder-decoder structures in total; fisheye images from n perspectives in a set of surround images are input into the same feature extraction module for parallel processing, the output of the feature extraction module is split into multiple perspectives and input into the distortion detail supplement module and the multi-perspective context fusion module for processing, and the obtained super multi-perspective features are input into the decoder to obtain a segmentation map.

4. The multi-view joint semantic segmentation method for surround fisheye images according to claim 2, characterized in that: The distortion detail supplement module is used to model the radial distortion of the fisheye image at the minimum cost, and use the dilation distortion information of the same object in the adjacent view to supplement the detail features in the current view; the detail features corresponding to each view are input into the deformable convolution Dconv, and the obtained distortion detail features are input into the auxiliary segmentation head to obtain the auxiliary segmentation mask And based on the auxiliary segmentation mask And the real mask under the i-th perspective calculates the segmentation loss to regularize and constrain the distorted detail features; The distortion detail features of the i-th viewing angle are completed pixel by pixel to obtain the completed distortion detail features.

5. The multi-view joint semantic segmentation method for surround fisheye images according to claim 4, characterized in that: The pixel-by-pixel completion of the distortion detail features of the i-th viewing angle to obtain the completed distortion detail features specifically includes: For each feature vector p on the distortion detail feature k , and perform feature correlation matching with the distortion detail features of adjacent perspectives to obtain a position weight matrix; then use the position weight matrix to perform weighted summation on the distortion detail features of adjacent perspectives, and after splicing with the original distortion detail features in the channel dimension, perform norm regularization and convolution to obtain the final completed distortion detail features.

6. The multi-view joint semantic segmentation method for surround fisheye images according to claim 2, characterized in that: The purpose of the multi-view context fusion module is to capture the context information missing from the current view in the adjacent views; First, the contextual features of the left and right adjacent views are evaluated Main perspective context features The correlation is calculated by the inner product between features. The correlation matrix is ​​used as an attention map to enhance the context features of each perspective, and the context features of each perspective enhanced by position correlation are obtained. The weights of the context features enhanced by position correlation of each view are calculated, and the weighted sum of the context features enhanced by position correlation of each view is calculated with the weights, and the complemented distortion detail features output by the feature fusion module and the distortion detail supplement module are merged, and the fusion features of each view obtained are channel-fused with the context features of the current view to obtain the super multi-view features corresponding to the current view 7. The multi-view joint semantic segmentation method for surround fisheye images according to claim 2, characterized in that: In step 3, the multi-view joint segmentation model is trained based on the training data set and the cross entropy loss, specifically including: Construct cross entropy loss l ce : Among them, P i represents the segmentation map output by the multi-view joint segmentation model for the fisheye image of the i-th view, and N is the batch number of the data group; The overall loss L of the multi-view joint segmentation model is constructed based on the cross entropy loss: Among them, W is the model weight, α represents the weight of auxiliary detail loss, and Dconv represents deformable convolution.

Images