Satellite and visual multi-modal sea ice perception fusion method and system based on bird's-eye view prior

Abstract

The application discloses a satellite and visual multi-modal sea ice perception fusion method and system based on an aerial view prior, and the method comprises the following steps: acquiring a sea ice satellite map based on a world geodetic system; converting the sea ice satellite map into a first feature map containing rich spatial context information, and extracting a second feature map based on the sea ice satellite map; fusing the first feature map and the second feature map to obtain a fusion feature map; and decoding and converting the fusion feature map to obtain a final sea ice map.

Description

Technical Field

[0001] This invention belongs to the field of multimodal sensor information fusion technology, specifically relating to a satellite and visual multimodal sea ice perception fusion method and system based on bird's-eye view prior. Background Technology

[0002] Against the backdrop of global climate change, the commercial value of Arctic shipping routes is becoming increasingly prominent. However, the dynamic and complex distribution of sea ice in the polar environment poses a severe challenge to navigation safety. In order to ensure the safe and efficient passage of ships through ice-covered areas, intelligent shipping systems must possess the ability to accurately and in real-time perceive the surrounding sea ice environment.

[0003] Currently, polar vessels mainly rely on two information sources to obtain information on sea ice distribution: 1. Satellite Remote Sensing Data: Satellite sensors such as Synthetic Aperture Radar (SAR) can acquire large-scale sea ice distribution maps. Its advantages include wide coverage and the ability to provide a global view of the sea ice situation. However, its disadvantages are also significant: First, the spatial resolution is relatively low, resulting in coarse outlines and edge information of the sea ice, making it difficult to identify small ice floes; second, the data update frequency is limited, typically on an hourly or daily basis, which cannot meet the needs of ships for close-range, highly dynamic decision-making.

[0004] 2. Shipborne Visual Perception: Using cameras mounted high on the ship's hull, combined with a deep learning semantic segmentation model, sea ice close to the ship can be identified in real time. Its advantages include the ability to acquire high-resolution images, detailed delineation of sea ice edges, and greater sensitivity to small ice floes. Its disadvantages are: firstly, the perception range is limited by the camera's field of view and effective range, unable to provide long-distance information, and is easily obstructed by the ship's structure; secondly, neural network-based recognition algorithms have the potential for false detections (such as misidentifying waves as ice) and missed detections, meaning their reliability is not 100%.

[0005] Most existing fusion strategies remain at the post-fusion or object-level fusion level, that is, extracting the polygonal contours of sea ice from two different data sources and then associating them through geometric matching (such as calculating centroid distance, overlap area, etc.). Such methods have significant technical bottlenecks: they heavily rely on the accuracy of upstream segmentation and detection algorithms. If the segmentation result of either data source is incorrect, subsequent geometric matching becomes extremely difficult or even fails, making it difficult to guarantee the robustness and accuracy of the fusion result.

[0006] Therefore, how to effectively integrate macroscopic and low-frequency information from satellite remote sensing with local and high-frequency information from shipborne vision to generate a unified, accurate, and reliable sea ice map that serves route planning is a key technical challenge that urgently needs to be solved in the field of polar intelligent shipping. Summary of the Invention

[0007] This invention aims to address the shortcomings of existing technologies and provides the following solutions: A method for fusing satellite and visual multimodal sea ice perception based on prior bird's-eye view includes the following steps: Sea ice satellite maps were obtained based on the World Geodetic System. The sea ice satellite map is converted into a first feature map containing rich spatial context information, and a second feature map is extracted based on the sea ice satellite map; The first feature map and the second feature map are fused to obtain a fused feature map; The fused feature map is decoded and transformed to obtain the final sea ice map.

[0008] Preferably, the method for obtaining the sea ice satellite map includes: Acquire satellite remote sensing data of sea ice and shipborne multi-view camera image data; By performing coordinate transformation on the satellite remote sensing sea ice data and the shipborne multi-view camera image data, the registration values ​​of the satellite data and camera data are unified in a coordinate system to obtain the sea ice satellite map.

[0009] Preferably, the method for obtaining the first feature map includes: The sea ice satellite map is input into the prior encoder network to obtain the first feature map: in, Fsat _ beer Represents the first feature map, Φ enc This represents the prior encoder network. Msat This represents a satellite map of sea ice.

[0010] Preferably, the method for obtaining the second feature map includes: For each pixel in the sea ice satellite map, a feature vector is extracted using a semantic segmentation network: in, c.p. Φ represents the eigenvector. seg This represents a semantic segmentation network. p Represents pixels; The probability distribution of each pixel across all possible depths is obtained through a depth prediction network: in, αp Represents a probability distribution. softmax This represents the activation function. fdepth This represents a deep prediction network. I Represents an image; Based on the feature vector and the probability distribution, a 3D point is created for each pixel and each possible depth. Cp , i The 3D coordinates of the 3D point are obtained using the camera intrinsic parameter matrix: in, Cp , i This represents the created 3D point. ap,i Represents pixels i The probability distribution, PCam , i Represents pixels i The corresponding 3D coordinates of the 3D point yes Represents pixels i Possible depth, K Represents the camera intrinsic parameter matrix, ( u , v () represents the coordinates of a 2D pixel; Based on the 3D coordinates, the 3D points are transformed from the camera coordinate system to the ship-centered ENU system using the camera extrinsic parameter matrix, resulting in... ego coordinate: in, Pego express ego coordinate, Tk Represents the camera extrinsic parameter matrix. PCam Represents 3D coordinates; Based on the above ego Coordinates are used to obtain the second feature map. Fcam _ beer .

[0011] Preferably, the method for obtaining the fused feature map includes: The first feature map and the second feature map are weighted based on a cross-attention mechanism to obtain a context feature matrix. Fcontext ; Based on the context feature matrix Fcontext The fused feature map is obtained as follows: in, Ffused Represents the fused feature map. Fcam represents the camera feature map, and FFN represents the location feedforward network.

[0012] Preferably, the method for obtaining the final sea ice map includes: decoding the fused feature map using a decoder to obtain a log-odds map. in, L Represents a logarithmic probability map, Ψ December Indicates the decoder network; After decoding by a multi-layer decoder, a multi-channel feature map is obtained. Dfinal ; Based on the feature map Dfinal The sea ice map obtained from a single channel is as follows: in, L’ Represents a sea ice map, Conv final This represents the final convolutional layer.

[0013] The present invention also provides a satellite and visual multimodal sea ice perception fusion system based on bird's-eye view priors. The system applies the above-mentioned method and includes: a satellite image acquisition module, a feature map extraction module, a feature fusion module, and a decoding and conversion module. The satellite image acquisition module acquires sea ice satellite maps based on the World Geodetic System. The feature map extraction module is used to convert the sea ice satellite map into a first feature map containing rich spatial context information, and extract a second feature map based on the sea ice satellite map; The feature fusion module is used to fuse the first feature map and the second feature map to obtain a fused feature map; The decoding and conversion module is used to decode and convert the fused feature map to obtain the final sea ice map.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention overcomes the shortcomings of existing object-level post-fusion techniques, which heavily rely on single-modal upstream segmentation results and suffer from poor robustness. It unifies the registration of macroscopic low-frequency satellite remote sensing sea ice data with local high-frequency shipborne visual data into a bird's-eye view space through coordinate system transformation. Addressing the depth ambiguity problem in visual perception, this invention does not directly predict depth but instead predicts the discrete depth distribution probability of pixels and constructs a 3D point cloud, thereby more accurately converting 2D images into 3D features. Based on this, the system utilizes a cross-attention mechanism to deeply fuse global satellite context and high-resolution local visual information at the feature level, achieving complementary advantages. Finally, the decoding network outputs a sea ice map in logarithmic probability form, intuitively reflecting the confidence level of sea ice occupation in each area, providing unified, accurate, and highly robust data support for path planning and dynamic safety decision-making for polar intelligent vessels. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of coordinate system transformation according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the cross-attention mechanism in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Example 1 In this embodiment, as Figure 1 As shown, a satellite and visual multimodal sea ice perception fusion method based on bird's-eye view priors includes the following steps: S1. Obtain satellite maps of sea ice based on the World Geodetic System.

[0020] The methods for obtaining sea ice satellite maps include: acquiring satellite remote sensing sea ice data and shipborne multi-view camera image data; and transforming the satellite remote sensing sea ice data and shipborne multi-view camera image data to unify the registration values ​​of the satellite data and camera data into a unified coordinate system to obtain the sea ice satellite map.

[0021] In this embodiment, as Figure 2 As shown, the coordinates include: World Geodetic System (WGS84), Earth-centered Earth-fixed (ECEF) coordinate system, local northeast-sky coordinate system (ENU) centered on the ship's hull, the ship's coordinate system, and the camera coordinate system. Satellite remote sensing sea ice data is a SAR sea ice classification map based on the WGS84 coordinate system. The ship's real-time geographical location can be obtained through a shipborne high-precision GNSS / IMU. .

[0022] WGS84 is a geodetic reference system, not a single coordinate system. It provides a unified reference framework for geospatial data worldwide, and is the reference system used by the GPS global positioning system. It approximates the Earth's physical surface using an idealized, regular rotating ellipsoid, and its commonly used coordinate representation is geodetic coordinates, i.e., longitude. l "latitude" ϕ "and the height of the ellipsoid" h Longitude l "Indicates the angle between the meridian plane and the prime meridian at a given point, latitude." ϕ "This represents the angle between the normal to a point on the ellipsoid and the equatorial plane, and the height of the ellipsoid." h " represents the distance from a point along the ellipsoid normal to the surface of the ellipsoid.

[0023] ECEF is a three-dimensional Cartesian coordinate system with its origin at the Earth's center of mass, rotating with the Earth, hence the name "Earth-fixed". Its Z-axis points to the Earth's North Pole as defined by WGS84, its X-axis points to the intersection of the Prime Meridian and the equator, and its Y-axis, together with the X and Z axes, forms a right-handed coordinate system, pointing to the intersection of 90 degrees East longitude and the equator. ECEF is well-suited for calculating spatial distances and vectors, but its description of directions is not intuitive.

[0024] ENU is a local, local Cartesian coordinate system. It is commonly used to describe the position and orientation of an object relative to a specific observation point. Because its coordinate axes directly correspond to the intuitive directions of "east," "north," and "up," it is widely used in navigation and autonomous driving. Its origin can be any reference point; in this invention, it is set as the ship's hull. The E-axis points due east, the N-axis points due north, and the U-axis is perpendicular to the local ground plane and points upwards. ENU is a tangent-plane coordinate system, and its effective range is typically limited to a few kilometers around the reference point because the Earth's curvature becomes non-negligible over a larger area.

[0025] The core conversion path in this invention is: WGS84 → ECEF → ENU.

[0026] First, WGS84 is converted to ECEF, which maps the latitude, longitude, and altitude of a point to three-dimensional Cartesian coordinates. Then, the radius of curvature of the trochanteric orbit is calculated. : in, a Represents the semi-major axis of the ellipsoid. e This represents the first eccentricity of the ellipsoid. The conversion formula can be obtained as follows: in,( X ,Y , Z ) represents the coordinates of the target point P.

[0027] Next is the conversion from ECEF to ENU. This conversion requires a reference point because ENU is a local coordinate system, and the reference point is chosen as the center of the ship. Suppose we want to calculate points... P ( X , Y , Z (At reference point) The coordinates are in the ENU coordinate system with the origin at the origin. First, calculate the coordinates from the reference point. P.S. Point to target point P Representation of a vector in the ECEF coordinate system v : Use this vector v Rotate from the ECEF coordinate system to P.S. For an ENU coordinate system with the origin, this requires a rotation matrix R, which is derived from the reference point. P.S. Geodetic coordinates Decide: Each row of the matrix corresponds to the representation of the three basis vectors of the ENU coordinate system (East, North, and Celestial) in the ECEF coordinate system. The difference vector in the ECEF coordinate system (...) dX , dY , dZ Convert to ENU coordinates using the rotation matrix R: Expanding the matrix multiplication above, we can obtain the calculation formula for each component: The coordinate system transformation is now complete, and a satellite map of sea ice is obtained.

[0028] S2. Convert the sea ice satellite map into a first feature map containing rich spatial context information, and extract a second feature map based on the sea ice satellite map.

[0029] Methods for obtaining the first feature map include: In step 1, a rasterized satellite map of sea ice aligned with the ship's coordinate system was obtained. Msat This map can be viewed as a single-channel two-dimensional matrix or tensor with dimensions of . H × W ×1: Where H and W represent the height and width of the BEV raster map (e.g., 200×200 grids), respectively, and 1 indicates that this is a single-channel input. Each element in the matrix... my This represents the corresponding grid in the BEV space. i , j The probability or concentration of sea ice, for example my =0.9 indicates that there is a 90% probability that the location is covered by sea ice.

[0030] Then, the sea ice satellite map Msat The input is fed into the prior encoder network, whose core objective is to transform a single-channel probability map with relatively simple semantic information. Msat This is transformed into a multi-channel first feature map containing rich spatial context information: in, Fsat _ beer Represents the first feature map, Φ enc This represents the prior encoder network. Msat This represents a satellite map of sea ice. The dimensionality of this output feature map needs to match the input dimensionality expected by subsequent fusion modules, specifically the number of channels C of the BEV feature map generated by the camera branch. The prior encoder network maintains a lightweight design, consisting of a series of 2D convolutional layers, normalization layers, and activation functions. Through end-to-end training, the encoder Φ... enc It will learn how to transform raw probability values ​​into feature representations most useful for subsequent fusion tasks. The final generated... Fsat _ beer Each element in the vector is described by a C-dimensional vector, which contains rich spatial semantic information about the point and its surrounding neighborhood.

[0031] In this embodiment, the method for obtaining the second feature map includes: Real-time camera vision feature generation. This step aims to convert perspective camera images into information-rich BEV feature maps. The Lift-Splat-Shoot paradigm is selected to process each input image from the surround-view camera; the core of this paradigm consists of the Lift and Splat steps.

[0032] Traditional methods struggle to accurately extract depth information from a single 2D image. "Lift" doesn't directly predict the precise depth value of each pixel; instead, it predicts a "discrete depth distribution probability" for each pixel along the direction of the camera's ray. Specifically, for each input image from the surround-view camera, a semantic segmentation network Φ... seg For each pixel on the sea ice satellite map Extract feature vectors c.p. : in, c.p. Φ represents the eigenvector. seg This represents a semantic segmentation network. p Represents pixels.

[0033] Next, the model generates a series of hypothetical depth points along the camera ray direction for each pixel, and passes them through a small depth prediction network. fdepth For each pixel p Predict an all possible depth Probability distribution at all possible depths: The probability distribution of each pixel at all possible depths is obtained through a depth prediction network. in, αp Representing the probability distribution, it is a D-dimensional vector. and satisfy , softmax This represents the activation function. fdepth This represents a deep prediction network. I Represents an image; Finally, feature and depth fusion is performed, based on feature vectors and probability distributions, for each pixel. p and every possible depth Create a 3D point Cp , i The feature of this point is its 2D feature. c.p. Rather than the probability at that depth αp , i The result of multiplication: Simultaneously utilize the camera intrinsic parameter matrix K 2D pixels ( u , v ) along with its depth yes Projecting the data back onto the camera's own 3D coordinate system yields the 3D coordinate points: in, Cp , i This represents the created 3D point. ap,i Represents pixels i The probability distribution, PCam , i Represents pixels i The corresponding 3D coordinates of the 3D point yes Represents pixels i Possible depth, K Represents the camera intrinsic parameter matrix, ( u ,v () represents the coordinates of a 2D pixel; thus, for each camera, a set of H×W×D points is obtained, each point represented by a 3D coordinate. PCam and a weighted eigenvector C constitute.

[0034] After the "Lift" step, each camera acquires a 3D point cloud with features. Currently, the coordinate systems of these point clouds are based on their respective cameras and cannot be used directly. The goal of "Splat" is to uniformly "flatten" these scattered 3D information from different perspectives into a bird's-eye view grid centered on the ship within the ENU coordinate system. First, for all 3D points generated by each camera... PCam Use the extrinsic matrix corresponding to each camera Tk Transform the points from the camera coordinate system to the ENU system centered on the ship (i.e., the Ego coordinate system): in, Pego express ego coordinate, Tk Represents the camera extrinsic parameter matrix. PCam Represents 3D coordinates; Then, these points are "flattened" onto the BEV grid, for each cell ( x , y ), its final feature vector F ( x , y The sum of the eigenvectors of all 3D points projected into this cell is: in, Indicates source from camera k The j The coordinates of a point in the vehicle coordinate system Indicates extraction of 3D points Pego of( x , y ) coordinates and map them to the indices of the BEV mesh, Indicates if Falling into the grid cell ( x , y If the value is within the range of ), then the value is 1; otherwise, it is 0.

[0035] The "Splat" step yields an information-rich second feature map of BEV with dimensions "C×X×Y". Fcam _ beer This image unifies the visual information from all cameras and encodes the geometric structure and semantic features of the scene.

[0036] S3. Fuse the first feature map and the second feature map to obtain the fused feature map.

[0037] In this embodiment, the method for obtaining the fused feature map includes: The first and second feature maps are weighted based on a cross-attention mechanism to obtain the context feature matrix. Fcontext Specifically: such as Figure 3 As shown, the feature fusion method is based on a cross-attention mechanism and is divided into query... Q ,key K Sum V , among which query Q Originating from camera feature maps Fcam _ beer Each query represents a specific location in the real-time camera view, posing the question: "For this point currently observed by the camera, what information in the global satellite map is most relevant to it?". K Originating from satellite feature maps Fsat _ beer Each key represents a location on the satellite map, holding a "title" or "label" for that location information, which is used in conjunction with queries. Q Compare and match. Values V Also derived from satellite feature maps, each value represents rich, real-world feature information about a specific location on the satellite map. Once a clear... QK Once matched, this information is passed to the camera features.

[0038] The original feature map is not used directly, but is linearly projected onto a learnable weight matrix. Q , K , V This three-space architecture enables the model to learn the most effective feature representations for attention tasks: in, It is a learnable weight matrix, obtained after projection. Q , K , V It is shaped as The matrix, where It is the dimension of the key / query vector.

[0039] Next, we calculate how much attention each camera feature pixel should "pay" to each satellite feature pixel: in, softmax The output of ( ) is the final attention weight matrix, which is then compared with the value matrix. VMultiplying them yields a weighted sum of all the value vectors, which is the final context feature matrix. .this Fcontext This will be used to update the camera's original features: Based on context feature matrix Fcontext The fused feature map is obtained as follows: in, Ffused Represents the fused feature map. Fcam represents the camera feature map, and FFN represents the location feedforward network.

[0040] S4. Decode and transform the fused feature map to obtain the final sea ice map.

[0041] In this embodiment, the method for obtaining the final sea ice map includes: First, a feature decoder is used, whose input is the feature map from the fusion module. This feature map possesses rich semantic information, but its spatial resolution may be lower than the desired map resolution. The decoder's task is to upsample the feature map while simultaneously restoring the deep, abstract feature space to concrete semantic predictions. Here, we use the decoder part of U-Net, which consists of a series of transposed convolutions, upsampling layers, and skip connections. The decoder network is defined as Ψ. December The fused feature map is decoded using a decoder to obtain a log-odds map. in, L Represents a logarithmic probability map, Ψ December The decoder network consists of two layers that progressively upsample and fuse the feature maps from the previous step through skip connections. This allows for the combination of high-level semantic information with fine spatial details at the lower level, which helps to generate clearer sea ice outlines.

[0042] in, Di Indicates the first i The feature map after layer decoding, where ReLU represents the activation function, BatchNorm represents batch normalization, and Conv... i Indicates the first i The convolutional layer is a layer where Upsample indicates upsampling. Yes Indicates the first i Layer jump connections.

[0043] After decoding by a multi-layer decoder, a multi-channel feature map is obtained. Dfinal Based on feature maps DfinalThis yields a single-channel sea ice map: in, L’ Represents a sea ice map, Conv final This represents the final convolutional layer. This layer does not use any activation function; the network directly regresses a logarithmic raster map, which defines a given raster. mxy Probability of being occupied p ( mxy The probability of not being occupied is 1- p ( mxy The logarithm of the ratio of ) in, l ( mxy ) represents the logarithmic probability.

[0044] Finally, for visualization purposes or for use by downstream modules that require probability inputs, it can be obtained from... L’ Extract the probability map P: so, l ( mxy )>0 corresponds to p ( mxy A value greater than 0.5 indicates that the grid cell is more likely to be occupied by sea ice. l ( mxy )<0 corresponds to p ( mxy If the value is less than 0.5, it indicates that the grid is more likely to be open sea area; l ( mxy )=0 corresponds to p ( mxy =0.5 indicates that the state is unknown.

[0045] Example 2 In this embodiment, a satellite and visual multimodal sea ice perception fusion system based on bird's-eye view priors includes: a satellite image acquisition module, a feature map extraction module, a feature fusion module, and a decoding and conversion module.

[0046] The satellite image acquisition module acquires sea ice satellite maps based on the World Geodetic System; the feature map extraction module converts the sea ice satellite maps into a first feature map containing rich spatial context information, and extracts a second feature map based on the sea ice satellite maps; the feature fusion module fuses the first and second feature maps to obtain a fused feature map; and the decoding and conversion module decodes and converts the fused feature map to obtain the final sea ice map.

[0047] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for fusing satellite and visual multimodal sea ice perception based on bird's-eye view priors, characterized in that, Includes the following steps: Sea ice satellite maps were obtained based on the World Geodetic System. The sea ice satellite map is converted into a first feature map containing rich spatial context information, and a second feature map is extracted based on the sea ice satellite map; The first feature map and the second feature map are fused to obtain a fused feature map; The fused feature map is decoded and transformed to obtain the final sea ice map.

2. The satellite and visual multimodal sea ice perception fusion method based on bird's-eye view prior as described in claim 1, characterized in that, The methods for obtaining the aforementioned sea ice satellite map include: Acquire satellite remote sensing data of sea ice and shipborne multi-view camera image data; By performing coordinate transformation on the satellite remote sensing sea ice data and the shipborne multi-view camera image data, the registration values ​​of the satellite data and camera data are unified in a coordinate system to obtain the sea ice satellite map.

3. The satellite and visual multimodal sea ice perception fusion method based on bird's-eye view prior as described in claim 1, characterized in that, The methods for obtaining the first feature map include: The sea ice satellite map is input into the prior encoder network to obtain the first feature map: in, Fsat _ bev Represents the first feature map, Φ enc This represents the prior encoder network. Msat This represents a satellite map of sea ice.

4. The satellite and visual multimodal sea ice perception fusion method based on bird's-eye view prior as described in claim 1, characterized in that, The methods for obtaining the second feature map include: For each pixel in the sea ice satellite map, a feature vector is extracted using a semantic segmentation network: in, cp Φ represents the eigenvector. seg This represents a semantic segmentation network. p Represents pixels; The probability distribution of each pixel across all possible depths is obtained through a depth prediction network: in, αp Represents a probability distribution. softmax This represents the activation function. fdepth This represents a deep prediction network. I Represents an image; Based on the feature vector and the probability distribution, a 3D point is created for each pixel and each possible depth. Cp , i The 3D coordinates of the 3D point are obtained using the camera intrinsic parameter matrix: in, Cp , i This represents the created 3D point. αp,i Represents pixels i The probability distribution, Pcam , i Represents pixels i The corresponding 3D coordinates of the 3D point di Represents pixels i Possible depth, K Represents the camera intrinsic parameter matrix, ( u , v () represents the coordinates of a 2D pixel; Based on the 3D coordinates, the 3D points are transformed from the camera coordinate system to the ship-centered ENU system using the camera extrinsic parameter matrix, resulting in... ego coordinate: in, Pego express ego coordinate, Tk Represents the camera extrinsic parameter matrix. Pcam Represents 3D coordinates; Based on the above ego Coordinates are used to obtain the second feature map. Fcam _ bev .

5. The satellite and visual multimodal sea ice perception fusion method based on bird's-eye view prior as described in claim 1, characterized in that, The method for obtaining the fused feature map includes: The first feature map and the second feature map are weighted based on a cross-attention mechanism to obtain a context feature matrix. Fcontext ; Based on the context feature matrix Fcontext The fused feature map is obtained as follows: in, Ffused Represents the fused feature map. Fcam represents the camera feature map, and FFN represents the location feedforward network.

6. The satellite and visual multimodal sea ice perception fusion method based on bird's-eye view prior as described in claim 1, characterized in that, The methods for obtaining the final sea ice map include: The fused feature map is decoded using a decoder to obtain a log-odds map: in, L Represents a logarithmic probability map, Ψ dec Indicates the decoder network; After decoding by a multi-layer decoder, a multi-channel feature map is obtained. Dfinal ; Based on the feature map Dfinal The sea ice map obtained from a single channel is as follows: in, L’ Represents a sea ice map, Conv final This represents the final convolutional layer.

7. A satellite and visual multimodal sea ice perception fusion system based on bird's-eye view priors, wherein the system applies the method described in any one of claims 1-6, characterized in that, include: Satellite image acquisition module, feature map extraction module, feature fusion module, and decoding and conversion module; The satellite image acquisition module acquires sea ice satellite maps based on the World Geodetic System. The feature map extraction module is used to convert the sea ice satellite map into a first feature map containing rich spatial context information, and extract a second feature map based on the sea ice satellite map; The feature fusion module is used to fuse the first feature map and the second feature map to obtain a fused feature map; The decoding and conversion module is used to decode and convert the fused feature map to obtain the final sea ice map.

Images