Roll-on passenger ship cabin high dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning method

Abstract

The application provides a high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning method in a passenger-rolling cabin, relates to the cabin positioning technical field, and comprises the following steps: cabin structure data, ship body six-degree-of-freedom motion parameters and multi-source perception data are acquired; the perception data is subjected to ship body motion decoupling processing; a multi-modal feature correspondence relationship is established; the confidence weight of each perception source is calculated; physically unreachable positioning solutions are removed according to cabin structure constraints and weighted fusion is carried out; and a three-dimensional positioning result is generated in combination with a deck topological structure. The application effectively overcomes the interference of a ship dynamic environment, and improves the multi-source heterogeneous perception fusion positioning precision and reliability.

Description

Technical Field

[0001] This invention relates to the field of ship cabin positioning technology, and in particular to a multi-source heterogeneous perception fusion positioning method for high dynamic three-dimensional scenes inside passenger roll-on / roll-off ships. Background Technology

[0002] With the continuous development of maritime tourism and passenger transport, the safety and service quality of ro-ro passenger ships, as an important means of water transportation, are receiving increasing attention. The interior of ro-ro passenger ships is typically complex, containing multiple decks and various functional areas, with dense crowds and frequent activity. To improve passenger safety, optimize emergency response capabilities, and enhance ship operation and management efficiency, accurate positioning of target objects (such as personnel and equipment) within the ship's interior is crucial. However, in the environment of a ro-ro passenger ship, factors such as metal structure obstruction, electromagnetic interference, personnel movement, and the ship's own motion make it difficult for a single sensing technology to provide stable and reliable positioning services. Currently, indoor positioning technologies such as WiFi, Bluetooth, and UWB are widely used in land-based buildings, but certain challenges remain in the unique environment of ro-ro passenger ships. Summary of the Invention

[0003] This invention provides a multi-source heterogeneous perception fusion and positioning method for high dynamic three-dimensional scenes inside passenger roll-on / roll-off ships, which can solve the problems in the prior art.

[0004] A first aspect of this invention provides a multi-source heterogeneous perception fusion and localization method for high-dynamic three-dimensional scenes inside a passenger roll-on / roll-off ship cabin, comprising: Acquire cabin structure data of passenger roll-on / roll-off ships, six-degree-of-freedom motion parameters of the hull, and multi-source spatial perception data of target objects; Based on the six-degree-of-freedom motion parameters of the hull, the rigid motion components of the hull are decomposed, and the multi-source spatial sensing data is synchronously compensated to eliminate the coupling effect of the hull motion, thus obtaining the multi-source spatial sensing data after the hull motion is decoupled. Feature extraction is performed on the decoupled multi-source spatial sensing data to establish the spatiotemporal correspondence between different sensing data sources and obtain an aligned multimodal feature set. Based on the multimodal feature set, the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source are identified, and the confidence weight of each sensing data source is calculated. Based on the multimodal feature set, candidate localization solutions for each sensing data source are generated. According to the spatial connectivity constraints of the cabin structure data, physically inaccessible candidate localization solutions are eliminated. The solutions are then weighted and fused together with the confidence weights to obtain the fused localization result of the target object. The fused positioning results are matched with the topology of the deck inside the cabin, and the target objects of different deck layers are distinguished by the vertical dimension floor constraints to generate a three-dimensional positioning result containing deck layer identifiers.

[0005] Based on the six-degree-of-freedom motion parameters of the hull, the rigid motion components of the hull are decomposed, and the multi-source spatial sensing data is synchronously compensated to eliminate the coupling effect of the hull motion, resulting in multi-source spatial sensing data after hull motion decoupling, including: The time series of roll angle, pitch angle and yaw angle in the six degrees of freedom motion parameters of the hull are transformed in the frequency domain. The angle frequency domain components in each angle time series that are within the range of the hull's natural rolling frequency are extracted. The angle frequency domain components are then subjected to inverse Fourier transform to obtain the hull rigid angle components. Based on the frequency characteristics of the rigid angle components of the hull, displacement frequency domain components with the same frequency are identified from the displacement time series of the six degrees of freedom motion parameters. The rigid displacement components of the hull are obtained by inverse Fourier transform. Combined with the rigid angle components of the hull, the rigid motion components of the hull are obtained. Extract the acquisition time of each data frame from the multi-source spatial sensing data, and find the corresponding rigid motion parameters in the rigid motion components of the hull at that time. The rigid motion parameters are converted into three-dimensional rotation transformation operators and three-dimensional translation transformation operators. The inverse transformation of the three-dimensional rotation transformation operator and the inverse transformation of the three-dimensional translation transformation operator are sequentially applied to the spatial sensing data in each data frame to generate multi-source spatial sensing data after decoupling of hull motion.

[0006] Feature extraction is performed on the decoupled multi-source spatial sensing data to establish spatiotemporal correspondences between different sensing data sources, resulting in an aligned multimodal feature set, including: The original modal features of each sensing data source are extracted from the decoupled multi-source spatial sensing data. The original modal features characterize the spatial distribution features and temporal evolution features of the target object. Obtain the timestamps, spatial coordinate system parameters, sampling frequency, and data format attributes of each sensing data source, and analyze the temporal and spatial heterogeneity of different sensing data sources. To address time domain heterogeneity, the sampling times of each sensing data source are mapped to a unified time base, and a time synchronization relationship across data sources is established through timestamp interpolation; to address spatial domain heterogeneity, the spatial coordinates of each sensing data source are uniformly transformed to the ship's coordinate system, and a spatial alignment relationship across data sources is established. Based on the temporal synchronization relationship and spatial alignment relationship, cross-data source features that are within the same time window and point to the same spatial region are identified, and different modal features that meet the spatiotemporal association conditions are associated and bound to generate an aligned multimodal feature set.

[0007] Based on the multimodal feature set, the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source are identified, and the confidence weight of each sensing data source is calculated, including: Based on the multimodal feature set, the spatial distribution features and temporal evolution features of each sensing data source representing the target object are extracted. Distance observation values ​​are extracted from the spatial distribution features. The shortest spatial connection path length is determined by combining the connectivity topology of the cabin structure, and the theoretical shortest propagation delay is calculated. The signal arrival time sequence is extracted from the time-series evolution characteristics, and the signal time of each sensing data source is mapped to a unified benchmark. When the actual propagation delay exceeds the propagation delay tolerance of the theoretical shortest propagation delay, an abnormal propagation path identifier is generated. Extract the time series of the received signal strength, analyze the rate of change of signal strength attenuation over time, calculate the theoretical signal strength attenuation value based on the material properties on the shortest spatial connection path, and mark the actual attenuation value as a signal strength attenuation indicator when it exceeds the attenuation tolerance of the theoretical attenuation value. The frequency of occurrence of the propagation path anomaly marker and the signal strength attenuation marker within a preset time window is statistically analyzed, converted into anomaly degree quantification values, and the confidence weight of each sensing data source is calculated.

[0008] Based on the multimodal feature set, candidate localization solutions from each sensing data source are generated. According to the spatial connectivity constraints of the cabin structure data, physically inaccessible candidate localization solutions are eliminated. These solutions are then weighted and fused using the confidence weights to obtain the fused localization result of the target object, including: Based on the multimodal feature set, the spatial distribution features and temporal evolution features of each sensing data source are extracted, the spatial distribution features of each sensing data source are converted into spatial position estimates in the ship coordinate system, and the search space range is determined. Combined with the motion parameters in the temporal evolution features, candidate localization solutions for each sensing data source are generated in the search space. Based on the cabin structure data, the spatial connectivity topology of the cabin is extracted, a cabin spatial connectivity graph is constructed, the spatial coordinates of each candidate positioning solution are mapped to the nodes of the cabin spatial connectivity graph, the spatial unit node to which each candidate positioning solution belongs is determined, and candidate positioning solutions belonging to physically unreachable isolated nodes are eliminated to obtain valid candidate positioning solutions. Within a continuous time window, the spatiotemporal trajectory of valid candidate localization solutions is extracted, and the spatial displacement at adjacent time points is calculated. When the displacement exceeds the motion range predicted based on temporal evolution features, it is marked as a motion anomaly candidate localization solution. Based on the confidence weight, the valid candidate localization solutions not marked as motion anomalies are weighted and fused to obtain the fused localization result of the target object.

[0009] The spatial distribution characteristics of each sensing data source are converted into spatial position estimates in the ship's coordinate system, and the search space range is determined. Combining the motion parameters in the temporal evolution characteristics, candidate localization solutions for each sensing data source are generated within the search space, including: The observation values ​​of each sensing data source are extracted from the spatial distribution characteristics. Based on the measurement model of each sensing data source, the observation covariance of the observation values ​​is calculated and mapped to the scale parameter of the search space. The observation values ​​are mapped to the ship coordinate system to generate the spatial position estimate of each sensing data source. Using the geometric center of the spatial position estimate of each sensing data source as the reference point, the boundary coordinates of the search space are determined according to the scale parameter to construct the candidate position search space. Motion parameters are extracted from the temporal evolution features. The motion parameters include velocity vectors and acceleration vectors. Based on the velocity vectors and acceleration vectors, the position offset of each sensing data source at the next moment is predicted. The position offset is superimposed on the spatial position estimate of each sensing data source to obtain the predicted position of each sensing data source. The predicted position is subjected to spatial constraint determination of the candidate position search space. When the predicted position is within the boundary coordinate range of the candidate position search space, the predicted position is used as a candidate positioning solution. When the predicted position exceeds the boundary coordinate range, the predicted position is projected onto the boundary, and the projected position is used as a candidate positioning solution.

[0010] The fused positioning results are matched with the topology of the decks inside the cabin. Vertical floor constraints are used to distinguish target objects on different deck levels, generating a 3D positioning result containing deck level identifiers, including: Extract the vertical coordinate range and inter-layer spacing information of each deck layer from the cabin structure data, construct a vertical dimension constraint model of the deck layer, obtain the three-dimensional spatial coordinates of the target object in the fusion positioning result, compare the vertical component of the three-dimensional spatial coordinates with the vertical dimension constraint model, and determine the candidate deck layer set to which the target object belongs. Based on the candidate deck layer set, extract the planar connectivity region and spatial accessibility information of the corresponding deck layer from the topology of the cabin deck, and establish the horizontal dimension topological constraints of the candidate deck layer. The horizontal coordinates of the target object in the fusion positioning result are projected onto the planar connected region corresponding to the candidate deck layer set. The spatial distance deviation between the projection point and the boundary of each connected region is calculated. Based on the spatial distance deviation and the horizontal dimension topological constraint, the target deck layer that matches the horizontal coordinates of the target object is selected from the candidate deck layer set. The hierarchical identifier of the target deck layer is associated and bound with the three-dimensional spatial coordinates in the fused positioning result to generate a three-dimensional positioning result containing the deck layer identifier.

[0011] A second aspect of the present invention provides a high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning system for passenger roll-on / roll-off ships, comprising: The first unit is used to acquire cabin structure data of passenger roll-on / roll-off ships, six-degree-of-freedom motion parameters of the hull, and multi-source spatial perception data of the target object; The second unit is used to decompose the rigid motion components of the hull based on the six degrees of freedom motion parameters of the hull, and to synchronously compensate the multi-source spatial sensing data to eliminate the coupling effect of the hull motion and obtain the multi-source spatial sensing data after the hull motion is decoupled. The third unit is used to extract features from the decoupled multi-source spatial sensing data, establish the spatiotemporal correspondence between different sensing data sources, and obtain an aligned multimodal feature set. The fourth unit is used to identify the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source based on the multimodal feature set, and to calculate the confidence weight of each sensing data source. The fifth unit is used to generate candidate localization solutions for each sensing data source based on the multimodal feature set, eliminate physically inaccessible candidate localization solutions according to the spatial connectivity constraints of the cabin structure data, and perform weighted fusion based on the confidence weight to obtain the fused localization result of the target object. The sixth unit is used to match the fused positioning results with the topology of the deck inside the cabin, use the vertical dimension floor constraints to distinguish target objects of different deck layers, and generate a three-dimensional positioning result containing deck layer identifiers.

[0012] A third aspect of the embodiments of the present invention, An electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0013] Fourth aspect of the present invention, A computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0014] The beneficial effects of this application are as follows: By decomposing and compensating the six degrees of freedom motion parameters of the hull, the interference of the high dynamic motion of the hull on the positioning accuracy is effectively eliminated, and the sensing data remains stable in the high dynamic ship environment.

[0015] By employing a feature extraction and spatiotemporal correspondence establishment method based on multi-source heterogeneous sensing data, effective alignment and fusion of data from different sensing technologies were achieved, overcoming the limitations of single sensing technologies in complex ship environments. The introduction of propagation path anomaly identification and signal strength attenuation identification mechanisms, combined with dynamic calculation of confidence weights, significantly improved the robustness of positioning in complex electromagnetic and multi-obstacle environments.

[0016] Based on spatial connectivity constraints derived from cabin structure data, physically unreachable candidate positioning solutions were effectively eliminated, resolving the fuzzy positioning problem inherent in traditional positioning methods within multi-deck structures. By differentiating different deck levels through vertical dimension floor constraints, true three-dimensional positioning was achieved, accurately identifying the deck level where the target object resides and improving the practicality of the positioning results in a three-dimensional ship environment. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the multi-source heterogeneous perception fusion and positioning method for high dynamic three-dimensional scenes inside the cabin of a passenger roll-on / roll-off ship, according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the method for generating multi-source spatial sensing data after decoupling from ship motion, as described in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0020] Figure 1 This is a flowchart illustrating the multi-source heterogeneous perception fusion and positioning method for high-dynamic three-dimensional scenes inside the cabin of a passenger roll-on / roll-off ship, as described in an embodiment of the present invention. Figure 1 As shown, the method includes: Acquire cabin structure data of passenger roll-on / roll-off ships, six-degree-of-freedom motion parameters of the hull, and multi-source spatial perception data of target objects; Based on the six-degree-of-freedom motion parameters of the hull, the rigid motion components of the hull are decomposed, and the multi-source spatial sensing data is synchronously compensated to eliminate the coupling effect of the hull motion, thus obtaining the multi-source spatial sensing data after the hull motion is decoupled. Feature extraction is performed on the decoupled multi-source spatial sensing data to establish the spatiotemporal correspondence between different sensing data sources and obtain an aligned multimodal feature set. Based on the multimodal feature set, the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source are identified, and the confidence weight of each sensing data source is calculated. Based on the multimodal feature set, candidate localization solutions for each sensing data source are generated. According to the spatial connectivity constraints of the cabin structure data, physically inaccessible candidate localization solutions are eliminated. The solutions are then weighted and fused together with the confidence weights to obtain the fused localization result of the target object. The fused positioning results are matched with the topology of the deck inside the cabin, and the target objects of different deck layers are distinguished by the vertical dimension floor constraints to generate a three-dimensional positioning result containing deck layer identifiers.

[0021] Figure 2 This is a flowchart illustrating the method for generating multi-source spatial sensing data after decoupling ship motion according to an embodiment of the present invention. In one optional implementation, based on the six-degree-of-freedom motion parameters of the ship, the rigid motion components of the ship are decomposed, and the multi-source spatial sensing data is synchronously compensated to eliminate the coupling effect of ship motion, thereby obtaining multi-source spatial sensing data after decoupling ship motion, including: The time series of roll angle, pitch angle and yaw angle in the six degrees of freedom motion parameters of the hull are transformed in the frequency domain. The angle frequency domain components in each angle time series that are within the range of the hull's natural rolling frequency are extracted. The angle frequency domain components are then subjected to inverse Fourier transform to obtain the hull rigid angle components. Based on the frequency characteristics of the rigid angle components of the hull, displacement frequency domain components with the same frequency are identified from the displacement time series of the six degrees of freedom motion parameters. The rigid displacement components of the hull are obtained by inverse Fourier transform. Combined with the rigid angle components of the hull, the rigid motion components of the hull are obtained. Extract the acquisition time of each data frame from the multi-source spatial sensing data, and find the corresponding rigid motion parameters in the rigid motion components of the hull at that time. The rigid motion parameters are converted into three-dimensional rotation transformation operators and three-dimensional translation transformation operators. The inverse transformation of the three-dimensional rotation transformation operator and the inverse transformation of the three-dimensional translation transformation operator are sequentially applied to the spatial sensing data in each data frame to generate multi-source spatial sensing data after decoupling of hull motion.

[0022] During ship navigation, it is necessary to collect the six degrees of freedom motion parameters of the hull, including the roll angle time series α(t), pitch angle time series β(t), bow angle time series γ(t), and the corresponding longitudinal displacement time series x(t), lateral displacement time series y(t), and vertical displacement time series z(t). These parameters can be continuously recorded using an onboard inertial measurement unit.

[0023] Frequency domain transformations were performed on these three sets of angular time series to extract the angular frequency components within the ship's natural rolling frequency range. Specifically, Fast Fourier Transform (FFT) was performed on α(t), β(t), and γ(t) to obtain their frequency domain representations α(f), β(f), and γ(f). The ship's natural rolling frequency is typically between 0.05Hz and 0.5Hz, varying depending on the ship's type and size. Therefore, extracting components within this frequency range from the frequency domain representation yields the frequency domain characteristics of the ship's rigid motion.

[0024] The extracted angular frequency domain components are transformed back to the time domain using inverse Fourier transform to obtain the hull rigidity angular components α_rigid(t), β_rigid(t), and γ_rigid(t). These components represent the swaying motion of the hull as a rigid body under the action of ocean waves, eliminating the influence of high-frequency vibrations and noise.

[0025] Based on the frequency characteristics of the rigid angular components of the hull, displacement frequency domain components of the same frequency are identified from the displacement time series of the six-degree-of-freedom motion parameters. Fast Fourier Transform is performed on the displacement time series x(t), y(t), and z(t) to obtain the frequency domain representations x(f), y(f), and z(f). Displacement frequency domain components are extracted within the same frequency range as the rigid angular components. Through inverse Fourier Transform, the rigid displacement components of the hull, x_rigid(t), y_rigid(t), and z_rigid(t), are obtained. Combined with the rigid angular components of the hull, a complete description of the rigid motion components of the hull is formed.

[0026] In the process of acquiring multi-source spatial sensing data, each data frame has a precise timestamp. The acquisition time t_frame of each data frame in the multi-source spatial sensing data is extracted, and the rigid motion parameters at the corresponding time are found in the rigid motion components of the hull. If the timestamp t_frame is not completely consistent with the sampling time of the rigid motion parameters, the rigid motion parameters at that time can be calculated by linear interpolation, namely α_rigid(t_frame), β_rigid(t_frame), γ_rigid(t_frame), x_rigid(t_frame), y_rigid(t_frame), and z_rigid(t_frame).

[0027] The rigid motion parameters are converted into three-dimensional rotation transformation operators and three-dimensional translation transformation operators. The three-dimensional rotation transformation operator can be represented by a rotation matrix, which is constructed based on Euler angles α_rigid(t_frame), β_rigid(t_frame), and γ_rigid(t_frame). The rotation matrix R can be obtained by a composite matrix of rotations around the z-axis, y-axis, and x-axis. The three-dimensional translation transformation operator is represented by the displacement vector T=[x_rigid(t_frame), y_rigid(t_frame), z_rigid(t_frame)].

[0028] For each frame of data in the multi-source spatial sensing data, which contains a 3D point cloud or spatial coordinate point P, the inverse transformation of the 3D rotation operator and the inverse transformation of the 3D translation operator are applied sequentially, specifically calculated as P'=R. (-1) ·(PT), where R (-1) The inverse of the rotation matrix is ​​actually equal to the transpose of the rotation matrix. Each spatial point is transformed into the ship's reference coordinate system, eliminating the influence of the ship's motion.

[0029] In practical applications, such as when an oceanographic survey vessel is equipped with a multibeam sonar system to conduct seabed topographic mapping, the swaying motion of the hull caused by the waves can lead to distortion and overlap of the sonar scan lines. By using the method described above, the data of each scan line can be transformed from the acquisition coordinate system to the hull reference coordinate system, correcting the geometric distortion caused by the hull motion, thereby obtaining a more accurate seabed topographic map.

[0030] Similarly, for 3D point cloud data of port facilities acquired by shipborne lidar systems, the above decoupling method can eliminate the influence of ship sway on the geometry of the point cloud, ensuring the spatial accuracy of the data. For image sequences acquired by shipborne camera systems, compensating for ship motion can improve the accuracy of image stitching and target recognition.

[0031] Through the above steps, the process of generating multi-source spatial sensing data after decoupling the ship's motion was completed, providing high-quality input for subsequent data analysis and processing. The data after decoupling the ship's motion can more accurately reflect the real spatial characteristics of the environment and improve the accuracy of marine exploration and environmental perception.

[0032] In one optional implementation, feature extraction is performed on the decoupled multi-source spatial sensing data to establish spatiotemporal correspondences between different sensing data sources, resulting in an aligned multimodal feature set, including: The original modal features of each sensing data source are extracted from the decoupled multi-source spatial sensing data. The original modal features characterize the spatial distribution features and temporal evolution features of the target object. Obtain the timestamps, spatial coordinate system parameters, sampling frequency, and data format attributes of each sensing data source, and analyze the temporal and spatial heterogeneity of different sensing data sources. To address time domain heterogeneity, the sampling times of each sensing data source are mapped to a unified time base, and a time synchronization relationship across data sources is established through timestamp interpolation; to address spatial domain heterogeneity, the spatial coordinates of each sensing data source are uniformly transformed to the ship's coordinate system, and a spatial alignment relationship across data sources is established. Based on the temporal synchronization relationship and spatial alignment relationship, cross-data source features that are within the same time window and point to the same spatial region are identified, and different modal features that meet the spatiotemporal association conditions are associated and bound to generate an aligned multimodal feature set.

[0033] In this specific embodiment, in the passenger roll-on / roll-off ship perception system, radar equipment is typically installed on the upper deck of the ship to ensure all-around coverage; optoelectronic equipment is distributed at key locations on each deck, focusing on monitoring passageways, hatches, and cargo areas; sonar equipment is installed on the hull for underwater environmental perception; and the automatic identification system antenna is installed on the top of the mast to ensure signal reception quality. These devices together constitute the ship's multi-source perception network, acquiring decoupled multi-source spatial perception data for raw modal feature extraction. For radar data, the range-azimuth detection algorithm extracts the reflection intensity, echo width, and Doppler velocity features of the target echo signal. These features characterize the electromagnetic reflection characteristics and relative motion state of the target object. For optoelectronic data, an image segmentation algorithm is applied to extract the shape contour features, texture features, and color features of the target area. These features characterize the visual appearance characteristics of the target object. For sonar data, acoustic features, including frequency distribution features, energy features, and sound source direction features, are extracted through acoustic spectrum analysis. These features characterize the acoustic radiation characteristics of the target object. For automatic identification system data, features such as ship identification, navigation status, and static information are extracted.

[0034] The extracted raw modal features collectively form the basis for representing the spatial distribution and temporal evolution characteristics of the target object. Regarding spatial distribution characteristics, radar-provided reflectance intensity distribution and range-azimuth data constitute the target's spatial outline; electro-optical data provides shape and color information that enhances spatial details; sonar data supplements underwater spatial information; and automatic identification system (AIS) data provides relative spatial positioning reference. By fusing these spatial distribution features, a comprehensive spatial perception of the target object in the passenger / roll-on / roll-off (Ro-Ro) vessel environment is formed. Regarding temporal evolution characteristics, continuously acquired radar Doppler velocity features characterize the target's trajectory; electro-optical data sequences reflect the time-varying characteristics of the target's appearance; temporal changes in sonar data reveal the underwater target's behavioral patterns; and dynamic information updates from the AIS reflect the navigation dynamics of surrounding vessels. These temporal evolution characteristics enable the positioning system to predict the target's future position, adapting to the highly dynamic environment of Ro-Ro vessels.

[0035] The system acquires timestamps, spatial coordinate system parameters, sampling frequencies, and data format attributes for each sensing data source to analyze the temporal and spatial heterogeneity of different sensing data sources. For radar data, it acquires the scanning cycle (typically 1-4 seconds / cycle), sampling timestamp, radar antenna installation position relative to the ship's hull, and radar coordinate system parameters. For optoelectronic equipment, it acquires the imaging frame rate (typically 25-30 frames / second), image acquisition timestamp, optoelectronic platform installation position, and rotation angle parameters. For sonar equipment, it acquires the sound wave transmission and reception cycle, signal processing timestamp, sonar array installation position relative to the ship's hull, and sonar coordinate system parameters. For automatic identification systems, it acquires the data update cycle (typically 2-10 seconds) and reception timestamp. Analysis of these parameters reveals temporal (inconsistent sampling frequencies, asynchronous time bases) and spatial (different sensor installation positions, inconsistent coordinate system definitions) heterogeneity among different data sources.

[0036] To address temporal heterogeneity, the sampling times of various sensing data sources are mapped to a unified time reference. A cross-data source temporal synchronization relationship is established through timestamp interpolation. The GPS time of the ship navigation system is selected as the unified time reference, and the timestamps of all data sources are converted to this time reference. For data with time delays, the time offset is calculated and corrected based on hardware processing delays and signal transmission delays. For data sources with inconsistent sampling frequencies, a time window segmentation method is adopted. An appropriate time window length (e.g., 1 second) is selected, and the sampling points of each data source within the window are mapped to a unified time at the point within the window using linear interpolation or spline interpolation. For periodically sampled data sources (e.g., radar rotating scans), the estimated value of the observed data at any time is calculated based on its periodic characteristics and scanning direction. Through these processes, unified alignment of different sensing data sources in the temporal dimension is achieved.

[0037] To address spatial heterogeneity, the spatial coordinates of various sensing data sources are uniformly transformed to the ship's coordinate system, establishing a spatial alignment relationship across data sources. The ship's coordinate system is defined, typically using the ship's center of gravity or a specific reference point as the origin. The bow direction is the positive x-axis, the starboard direction is the positive y-axis, and the vertically upward direction is the positive z-axis. Based on the installation position and attitude of each sensor, a coordinate transformation matrix is ​​established from the sensor's local coordinate system to the ship's coordinate system. For radar data, the target position represented in polar coordinates (range-azimuth) is converted to Cartesian coordinates, and then transformed to the ship's coordinate system using a coordinate transformation matrix. For photoelectric data, the target position in the image pixel coordinate system is transformed to the ship's coordinate system using camera intrinsic and extrinsic parameter matrices. For sonar data, the acoustic azimuth-elevation and ranging information are converted to a three-dimensional position in the ship's coordinate system. Through these coordinate transformations, unified alignment across different sensing data sources in spatial dimensions is achieved.

[0038] Based on the aforementioned temporal synchronization and spatial alignment relationships, cross-data source features that point to the same spatial region within the same time window are identified. Different modal features that meet the spatiotemporal association conditions are associated and bound to generate an aligned multimodal feature set. In specific implementation, the criteria for determining spatiotemporal association are defined as follows: in the time dimension, if the sampling time difference between two data sources is less than a preset threshold (e.g., 0.5 seconds), the time association condition is considered to be met; in the spatial dimension, if the Euclidean distance between the target spatial locations pointed to by the two data sources is less than a preset threshold (e.g., 10 meters), the spatial association condition is considered to be met. For different modal features that meet the spatiotemporal association conditions, a unique association identifier is generated, and these features are bound into a multimodal feature vector. For one-to-many or many-to-one association cases, nearest neighbor or probabilistic data association algorithms are used for optimization matching to generate an aligned multimodal feature set. Each set element contains feature combinations from different perceptual data sources but pointing to the same target object.

[0039] Through the above steps, feature extraction and spatiotemporal alignment across data sources of decoupled multi-source spatial sensing data were achieved, providing basic data support for subsequent multimodal fusion sensing.

[0040] In one optional implementation, based on the multimodal feature set, the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source are identified, and the confidence weight of each sensing data source is calculated, including: Based on the multimodal feature set, the spatial distribution features and temporal evolution features of each sensing data source representing the target object are extracted. Distance observation values ​​are extracted from the spatial distribution features. The shortest spatial connection path length is determined by combining the connectivity topology of the cabin structure, and the theoretical shortest propagation delay is calculated. The signal arrival time sequence is extracted from the time-series evolution characteristics, and the signal time of each sensing data source is mapped to a unified benchmark. When the actual propagation delay exceeds the propagation delay tolerance of the theoretical shortest propagation delay, an abnormal propagation path identifier is generated. Extract the time series of the received signal strength, analyze the rate of change of signal strength attenuation over time, calculate the theoretical signal strength attenuation value based on the material properties on the shortest spatial connection path, and mark the actual attenuation value as a signal strength attenuation indicator when it exceeds the attenuation tolerance of the theoretical attenuation value. The frequency of occurrence of the propagation path anomaly marker and the signal strength attenuation marker within a preset time window is statistically analyzed, converted into anomaly degree quantification values, and the confidence weight of each sensing data source is calculated.

[0041] In this specific embodiment, the spatial distribution features and temporal evolution features of each sensing data source representing the target object are extracted from the multimodal feature set. When extracting the spatial distribution features, the focus is on the distance observation values ​​representing the spatial position of the target object. For wireless signal systems, the distance is estimated based on the correspondence between the received signal strength indication value and the signal transmission source. For visual sensors, the depth information of the target in the image is calculated using the triangulation principle and converted into the actual distance value. The extracted distance observation values ​​are stored in vector form, containing a triplet of target ID, observation timestamp, and distance value.

[0042] When determining the shortest spatial connectivity path length by combining the connectivity topology of the cabin structure, a three-dimensional topology model of the passenger roll-on / roll-off ship cabin is established. This model divides the cabin into multiple interconnected nodes, where nodes represent cabins, passageways, or key locations, and edges represent direct connections between nodes. The weights of the edges are set according to the actual physical distance, while also considering the influence of factors such as passageway width and obstacle distribution on connectivity. The shortest path between the signal source and the receiving point is searched on the topology graph using breadth-first search, and the shortest spatial connectivity path length is obtained. This path length reflects the theoretical shortest distance for signal propagation within the cabin, providing a benchmark for subsequent propagation delay analysis.

[0043] When calculating the theoretical shortest propagation delay, different processing is performed according to the propagation characteristics of different types of signals. Based on the shortest spatial path length and the signal propagation speed (such as the speed of sound wave propagation of about 340 meters / second and the speed of electromagnetic wave propagation of about the speed of light), the theoretical shortest propagation delay is calculated. For example, if the shortest spatial path length is 15 meters, the theoretical shortest propagation delay for a sound wave sensor is about 0.044 seconds.

[0044] To extract the signal arrival time series from the temporal evolution characteristics, since different sensing data sources use different time bases, it is necessary to map the signal time of each sensing data source to a unified base. This can be done by using the system global clock or by selecting a specific event as a reference point for time calibration. The actual propagation delay is calculated, which is the time difference between the signal being emitted and received by the sensing data source. When the actual propagation delay exceeds the propagation delay tolerance of the theoretical minimum propagation delay, a propagation path anomaly flag is generated. The propagation delay tolerance can be set to 20% of the theoretical value. That is, for the example above, if the actual propagation delay exceeds 0.0528 seconds (0.044 × 1.2), it is marked as a propagation path anomaly.

[0045] Simultaneously, the time series of the received signal strength is extracted, and the rate of change of signal strength attenuation over time is analyzed. Based on the material properties along the shortest spatial connection path, such as the wall material and obstacle type along the path, the theoretical signal strength attenuation value is calculated. For example, for radio signals, the signal strength attenuates by about 6dB for every doubling of distance in free space, while an additional 15-20dB is attenuated through metal walls. When the actual attenuation value exceeds the attenuation tolerance of the theoretical attenuation value, it is marked as a signal strength attenuation indicator. The attenuation tolerance can be set to 30% of the theoretical attenuation value, or dynamically adjusted according to the complexity of the environment.

[0046] In practical applications, a sensing data source generates 100 data points within a 10-second sampling window. If 15 data points are found to have propagation path anomaly indicators and 12 data points are found to have signal strength attenuation indicators, then the anomaly rates within this window are 15% and 12%, respectively. The frequency of occurrence of propagation path anomaly indicators and signal strength attenuation indicators within a preset time window is statistically analyzed. The time window can be set as a sliding window, such as the most recent 10 seconds or the most recent 100 sampling points. The anomaly frequency is converted into an anomaly degree quantification value. The sigmoid function can be used to map the anomaly frequency to the [0,1] interval. For example, for propagation path anomalies, the anomaly degree quantification value can be calculated using 1 / (1+exp(-(abnormal frequency-0.2)*10)). The signal strength attenuation anomaly degree quantification value can be calculated using a similar method.

[0047] Based on the anomaly quantification value, the confidence weight of each sensing data source is calculated. The confidence weight is inversely proportional to the anomaly quantification value and can be expressed as 1 minus the weighted sum of the anomaly quantification values. The weighting coefficient can be adjusted according to the application scenario. For example, the propagation path anomaly weight is 0.6 and the signal strength attenuation anomaly weight is 0.4. For instance, if the propagation path anomaly quantification value of a certain sensing data source is 0.3 and the signal strength attenuation anomaly quantification value is 0.2, then its confidence weight is 1-(0.3×0.6+0.2×0.4)=0.74.

[0048] In practical applications, such as personnel positioning in a confined space, when a signal received by a certain sensing data source (such as a WiFi signal strength sensor) shows obvious abnormal propagation path, it is due to signal reflection or diffraction. In this case, the weight of that data source in the fusion positioning should be reduced. Similarly, when the target thermal signal intensity detected by the infrared thermal imaging sensor is abnormally attenuated, it is because there is an unrecorded obstruction between the target and the sensor. The reliability of the sensor data should be reduced accordingly.

[0049] The confidence weights calculated using the above method can be used for subsequent multi-source data fusion processing to achieve a more accurate estimation of the target object's location and improve the overall perception accuracy of the system.

[0050] In one optional implementation, candidate localization solutions for each sensing data source are generated based on the multimodal feature set. Physically inaccessible candidate localization solutions are eliminated according to the spatial connectivity constraints of the cabin structure data. These solutions are then weighted and fused using the confidence weights to obtain the fused localization result for the target object, including: Based on the multimodal feature set, the spatial distribution features and temporal evolution features of each sensing data source are extracted, the spatial distribution features of each sensing data source are converted into spatial position estimates in the ship coordinate system, and the search space range is determined. Combined with the motion parameters in the temporal evolution features, candidate localization solutions for each sensing data source are generated in the search space. Based on the cabin structure data, the spatial connectivity topology of the cabin is extracted, a cabin spatial connectivity graph is constructed, the spatial coordinates of each candidate positioning solution are mapped to the nodes of the cabin spatial connectivity graph, the spatial unit node to which each candidate positioning solution belongs is determined, and candidate positioning solutions belonging to physically unreachable isolated nodes are eliminated to obtain valid candidate positioning solutions. Within a continuous time window, the spatiotemporal trajectory of valid candidate localization solutions is extracted, and the spatial displacement at adjacent time points is calculated. When the displacement exceeds the motion range predicted based on temporal evolution features, it is marked as a motion anomaly candidate localization solution. Based on the confidence weight, the valid candidate localization solutions not marked as motion anomalies are weighted and fused to obtain the fused localization result of the target object.

[0051] In this specific embodiment, based on the multimodal feature set, the spatial distribution features of each sensing data source are mapped to a unified ship coordinate system through coordinate system transformation. For example, for thermal imaging data, the two-dimensional position of the hotspot area is converted into a three-dimensional spatial position estimate using known installation position parameters and field of view information; for camera data, monocular or binocular visual positioning methods are used, combined with known camera intrinsic and extrinsic parameters, to convert the target position in the image plane into a three-dimensional position estimate in the ship coordinate system; for acoustic data, based on the spatial layout of the microphone array, the azimuth angle and estimated distance of the sound source are converted into a spatial position estimate in the ship coordinate system. Taking the current spatial position estimate as the center, and combining the motion parameters in the temporal evolution features, a three-dimensional search sphere or ellipsoid is set. Within this search space, candidate positioning solutions are generated for each sensing data source through methods such as particle filtering or Monte Carlo sampling. These candidate positioning solutions represent the positional distribution of the target object.

[0052] Candidate positioning solutions are selected based on the cabin structure data. The cabin structure data is processed to extract structural elements such as walls, passages, doors and windows inside the cabin. A topological structure representing the connectivity of the cabin space is constructed. Specifically, the cabin space is discretized into grid cells. Each passable grid cell is used as a node in the spatial connectivity graph. Connection edges are established between adjacent passable cells to form the cabin space connectivity graph.

[0053] For each candidate location solution, its spatial coordinates are mapped to the nearest spatial cell node to determine the spatial cell to which it belongs. Using a breadth-first search or depth-first search algorithm, it is checked whether the spatial cell has a connected path to the known starting point or reference point. If no connected path exists, the candidate location solution is determined to belong to a physically unreachable isolated node and should be eliminated. For example, if a candidate location solution is located in an area completely isolated from the current known location by a wall and cannot be reached through doors, windows, or other passages, then the candidate location solution is eliminated. In this way, all the remaining candidate location solutions are physically reachable and valid candidate location solutions.

[0054] To further improve positioning accuracy, the spatiotemporal trajectories of valid candidate positioning solutions are extracted within a continuous time window. Specifically, valid candidate positioning solutions at adjacent time points are connected to form a spatiotemporal trajectory sequence. The spatial displacement at adjacent time points is calculated and compared with the motion range predicted based on temporal evolution characteristics. When the actual displacement significantly exceeds the predicted motion range (e.g., more than twice the predicted range), the candidate positioning solution is marked as a motion anomaly candidate positioning solution.

[0055] For valid candidate localization solutions not marked as motion anomalies, a weighted fusion is performed based on the previously determined confidence weights of each sensor data source. Specifically, the fusion method is to calculate a weighted average position. Assuming there are n valid candidate localization solutions, and the position coordinates of each localization solution are P... i The confidence weight of the corresponding perceived data source is w. i Then the fused target position P can be represented as the weighted average of all valid candidate localization solutions. In practical applications, the confidence weight will be dynamically adjusted according to environmental conditions. For example, the weight of video data will be increased in well-lit environments, while the weight of acoustic data will be decreased in noisy environments.

[0056] Through the above steps, combined with multimodal perception data, cabin structure constraints, and temporal evolution characteristics, accurate and reliable target object fusion positioning results are obtained, effectively solving the accuracy and reliability problems of target positioning in complex cabin environments. It can achieve stable tracking and precise positioning of targets under conditions of limited field of view and complex environment.

[0057] In one optional implementation, the spatial distribution characteristics of each sensing data source are converted into spatial position estimates in the ship's coordinate system, and the search space range is determined. Combining the motion parameters in the temporal evolution characteristics, candidate positioning solutions for each sensing data source are generated within the search space. This includes: extracting observation values ​​from each sensing data source from the spatial distribution characteristics; calculating the observation covariance of the observation values ​​according to the measurement model of each sensing data source and mapping it to the scale parameter of the search space; mapping the observation values ​​to the ship's coordinate system to generate spatial position estimates for each sensing data source; and determining the boundary coordinates of the search space based on the scale parameter, using the geometric center of the spatial position estimates of each sensing data source as the reference point, to construct the candidate position search space. Motion parameters are extracted from the temporal evolution features. The motion parameters include velocity vectors and acceleration vectors. Based on the velocity vectors and acceleration vectors, the position offset of each sensing data source at the next moment is predicted. The position offset is superimposed on the spatial position estimate of each sensing data source to obtain the predicted position of each sensing data source. The predicted position is subjected to spatial constraint determination of the candidate position search space. When the predicted position is within the boundary coordinate range of the candidate position search space, the predicted position is used as a candidate positioning solution. When the predicted position exceeds the boundary coordinate range, the predicted position is projected onto the boundary, and the projected position is used as a candidate positioning solution.

[0058] In this specific embodiment, observations from various sensing data sources are extracted from spatial distribution characteristics. These observations include target range, azimuth, and elevation angle information acquired by sensors such as radar, electro-optical, and sonar. Taking a radar sensor as an example, the observations include the target's slant range R, azimuth angle θ, and elevation angle φ. Based on the sensor installation location and attitude parameters, a measurement model is established, and the observations are converted into three-dimensional coordinates (x, y, z) in the ship's coordinate system.

[0059] Calculating the observation covariance is a crucial step in ensuring positional accuracy. The observation covariance reflects the measurement accuracy of each sensor and is typically related to the measured distance, angle, and the sensor's own errors. For example, the radar distance measurement error σ... R The azimuth measurement error σ is proportional to the measurement range. θ Related to the azimuth resolution, the observation covariance is mapped to the ship's coordinate system through coordinate transformation, forming an uncertainty ellipsoid for position estimation, with its semi-axis length serving as the scale parameter of the search space.

[0060] After obtaining the spatial location estimates of each sensing data source, their geometric center is calculated as the reference point of the search space. The coordinates of the geometric center are the arithmetic mean of the coordinates of each data source location. Combined with the aforementioned scale parameters, the boundary coordinates of the search space are determined. The boundary range is usually set to be centered on the reference point, with each dimension extended by 2-3 times the scale parameter, forming a cuboid search space.

[0061] The temporal evolution features provide target motion information, including velocity vectors and acceleration vectors. These parameters are obtained through differential calculation of multiple consecutive frames of observation data. For newly acquired observation data, the position of each sensing data source at the current time is predicted by combining the position estimate and calculated motion parameters from the previous moment. The prediction process adopts a uniformly accelerated motion model: new position = old position + velocity × time interval + 0.5 × acceleration × square of time interval.

[0062] Determining the boundary constraints of the search space for the predicted location is crucial. When the predicted location is within the boundary coordinate range, it is directly used as a candidate location solution. When the predicted location exceeds the boundary, boundary projection processing is required. The boundary projection adopts the nearest point projection method, which projects the predicted location that exceeds the boundary onto the boundary surface to obtain a candidate location that satisfies the spatial constraints.

[0063] Taking multi-sensor cooperative tracking at sea as an example, when sonar and radar simultaneously detect a target, the observations from both sensors are converted into position estimates in the ship's coordinate system. Sonar measurements are more accurate underwater, while radar is more accurate at targets above the water surface. By calculating the observation covariance, the scale parameter of the search space is determined. Sonar has a smaller scale parameter in the depth direction and a larger one in the horizontal direction; radar is the opposite. Combining the target's historical trajectory, velocity and acceleration information are extracted to predict the target's position at the current moment. If the target is performing a rapid turning maneuver, the predicted position will exceed the search space boundary. In this case, boundary projection processing is used to project the predicted position onto the search space boundary as the final candidate positioning solution.

[0064] In practical applications, when there are significant differences in the location estimates of multi-source data, a weighted average method can be used to determine the benchmark point in the search space, with the weights proportional to the observation accuracy of each sensor. By reasonably setting the search space range and scientifically handling the boundary constraints of the predicted location, the accuracy and stability of multi-source sensing data fusion positioning can be effectively improved.

[0065] In one optional implementation, the fused positioning result is matched with the topology of the deck within the cabin, and target objects on different deck levels are distinguished using vertical dimension floor constraints to generate a three-dimensional positioning result containing deck level identifiers, including: Extract the vertical coordinate range and inter-layer spacing information of each deck layer from the cabin structure data, construct a vertical dimension constraint model of the deck layer, obtain the three-dimensional spatial coordinates of the target object in the fusion positioning result, compare the vertical component of the three-dimensional spatial coordinates with the vertical dimension constraint model, and determine the candidate deck layer set to which the target object belongs. Based on the candidate deck layer set, extract the planar connectivity region and spatial accessibility information of the corresponding deck layer from the topology of the cabin deck, and establish the horizontal dimension topological constraints of the candidate deck layer. The horizontal coordinates of the target object in the fusion positioning result are projected onto the planar connected region corresponding to the candidate deck layer set. The spatial distance deviation between the projection point and the boundary of each connected region is calculated. Based on the spatial distance deviation and the horizontal dimension topological constraint, the target deck layer that matches the horizontal coordinates of the target object is selected from the candidate deck layer set. The hierarchical identifier of the target deck layer is associated and bound with the three-dimensional spatial coordinates in the fused positioning result to generate a three-dimensional positioning result containing the deck layer identifier.

[0066] In this specific embodiment, the vertical coordinate range and inter-layer spacing information of each deck layer are extracted from the cabin structure data to construct a vertical dimension constraint model for the deck layers. Specifically, the cabin structure data is parsed to extract the absolute ground height value and floor height information of each deck layer, forming a data structure such as {deck layer identifier, bottom height value, top height value}. For example, for a three-deck structure, we can obtain: Deck 1 {bottom height 0 meters, top height 2.8 meters}, Deck 2 {bottom height 3 meters, top height 5.8 meters}, and Deck 3 {bottom height 6 meters, top height 8.8 meters}. These data constitute the vertical dimension constraint model, which is used for subsequent deck layer determination.

[0067] After obtaining the three-dimensional spatial coordinates of the target object in the fusion positioning result, the vertical component of the coordinates is compared with the vertical dimension constraint model. Specifically, the z value in the three-dimensional coordinates (x, y, z) of the target object is extracted, and it is checked whether the z value falls within the height range of each deck layer. For example, if the vertical coordinate z of the target object is 5.2 meters, by comparing it with the vertical dimension constraint model, it can be preliminarily determined that the target is located on deck 2 (within the range of 3 meters to 5.8 meters). If the z value of the target is 2.9 meters, it is located in the transition area between deck 1 and deck 2. At this time, both adjacent deck layers need to be included as candidate layers in the candidate deck layer set.

[0068] Based on the candidate deck layer set determined in the previous step, the planar connectivity and spatial accessibility information of the corresponding deck layer are extracted from the topology of the cabin decks to establish horizontal topological constraints. For each candidate deck layer, its plan layout is extracted, the boundary contours of connected areas such as corridors, rooms, and open areas are identified, and the position information of connection points such as doors and passages is recorded. For example, for deck layer 2, the geometric boundaries of all its rooms, the center lines of the corridors, and the position coordinates of the doors are extracted to construct a connected graph structure representing the spatial connectivity between different areas within this layer.

[0069] The horizontal coordinates of the target object in the fused localization results are projected onto the planar connected regions corresponding to the candidate deck layers. Specifically, the horizontal component (x, y) of the target object's three-dimensional coordinates (x, y, z) is taken and projected onto the planar layout of each candidate deck layer to obtain the coordinates of the projection point. The spatial distance deviation between the projection point and the boundary of each connected region is calculated. For example, if the projection point of the target object on deck 2 is located inside a corridor and is 0.8 meters away from the corridor boundary, while the projection point on deck 1 is located inside a wall and is 1.5 meters away from the nearest passable area, then the target is located on deck 2.

[0070] Based on the calculated spatial distance deviation and horizontal dimensional topological constraints, the target deck layer that matches the horizontal coordinates of the target object is selected from the candidate deck layer set. A scoring mechanism is set up to calculate the matching score for each candidate layer, including: whether the projection point is located in the passable area, the minimum distance to the boundary of the inaccessible area, and the relative positional relationship with key reference objects (such as walls and doors). The candidate layer with the highest matching score is selected as the target deck layer. If the scores are close, the confidence of the vertical coordinates or the continuity of the historical positioning trajectory can be combined for further judgment.

[0071] The target deck level identifier is associated with the 3D spatial coordinates in the fused positioning result to generate a 3D positioning result containing the deck level identifier. Specifically, the determined deck level identifier (e.g., "Deck 2") is added to the attribute information of the target object, together with the original 3D coordinates (x, y, z), to form enhanced positioning information {object ID, deck level identifier, x coordinate, y coordinate, z coordinate, timestamp}. This result not only preserves the original accurate spatial coordinates but also provides semantic deck level information, facilitating subsequent applications such as cabin navigation and personnel monitoring.

[0072] In practical applications, the positioning of personnel inside a large ship can simultaneously track multiple target objects. For example, if ten crew members are detected to be distributed on different decks, the above method can accurately identify the specific deck where each crew member is located and display it in different colors in the three-dimensional model of the ship management system. Even in the stairwell or elevator area between decks, the transition state of the target can be accurately determined based on the vertical coordinates and horizontal topological constraints, thereby improving the accuracy and reliability of personnel positioning inside the cabin.

[0073] When a target object moves between decks, its three-dimensional coordinate changes can be continuously tracked and deck markings updated in real time to create a complete movement trajectory record. This is of great value for emergency evacuation, personnel management, and ship safety monitoring, enabling the rapid location of specific personnel and the planning of optimal evacuation routes in emergency situations.

[0074] This invention relates to a high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning system for passenger roll-on / roll-off ships, comprising: The first unit is used to acquire cabin structure data of passenger roll-on / roll-off ships, six-degree-of-freedom motion parameters of the hull, and multi-source spatial perception data of the target object; The second unit is used to decompose the rigid motion components of the hull based on the six degrees of freedom motion parameters of the hull, and to synchronously compensate the multi-source spatial sensing data to eliminate the coupling effect of the hull motion and obtain the multi-source spatial sensing data after the hull motion is decoupled. The third unit is used to extract features from the decoupled multi-source spatial sensing data, establish the spatiotemporal correspondence between different sensing data sources, and obtain an aligned multimodal feature set. The fourth unit is used to identify the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source based on the multimodal feature set, and to calculate the confidence weight of each sensing data source. The fifth unit is used to generate candidate localization solutions for each sensing data source based on the multimodal feature set, eliminate physically inaccessible candidate localization solutions according to the spatial connectivity constraints of the cabin structure data, and perform weighted fusion based on the confidence weight to obtain the fused localization result of the target object. The sixth unit is used to match the fused positioning results with the topology of the deck inside the cabin, use the vertical dimension floor constraints to distinguish target objects of different deck layers, and generate a three-dimensional positioning result containing deck layer identifiers.

[0075] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0076] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0077] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source heterogeneous perception fusion and positioning method for high-dynamic three-dimensional scenes inside the cabin of a passenger roll-on / roll-off ship, characterized in that, include: Acquire cabin structure data of passenger roll-on / roll-off ships, six-degree-of-freedom motion parameters of the hull, and multi-source spatial perception data of target objects; Based on the six-degree-of-freedom motion parameters of the hull, the rigid motion components of the hull are decomposed, and the multi-source spatial sensing data is synchronously compensated to eliminate the coupling effect of the hull motion, thus obtaining the multi-source spatial sensing data after the hull motion is decoupled. Feature extraction is performed on the decoupled multi-source spatial sensing data to establish the spatiotemporal correspondence between different sensing data sources and obtain an aligned multimodal feature set. Based on the multimodal feature set, the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source are identified, and the confidence weight of each sensing data source is calculated. Based on the multimodal feature set, candidate localization solutions for each sensing data source are generated. According to the spatial connectivity constraints of the cabin structure data, physically inaccessible candidate localization solutions are eliminated. The solutions are then weighted and fused together with the confidence weights to obtain the fused localization result of the target object. The fused positioning results are matched with the topology of the deck inside the cabin, and the target objects of different deck layers are distinguished by the vertical dimension floor constraints to generate a three-dimensional positioning result containing deck layer identifiers.

2. The method according to claim 1, characterized in that, Based on the six-degree-of-freedom motion parameters of the hull, the rigid motion components of the hull are decomposed, and the multi-source spatial sensing data is synchronously compensated to eliminate the coupling effect of the hull motion, resulting in multi-source spatial sensing data after hull motion decoupling, including: The time series of roll angle, pitch angle and yaw angle in the six degrees of freedom motion parameters of the hull are transformed in the frequency domain. The angle frequency domain components in each angle time series that are within the range of the hull's natural rolling frequency are extracted. The angle frequency domain components are then subjected to inverse Fourier transform to obtain the hull rigid angle components. Based on the frequency characteristics of the rigid angle components of the hull, displacement frequency domain components with the same frequency are identified from the displacement time series of the six degrees of freedom motion parameters. The rigid displacement components of the hull are obtained by inverse Fourier transform. Combined with the rigid angle components of the hull, the rigid motion components of the hull are obtained. Extract the acquisition time of each data frame from the multi-source spatial sensing data, and find the corresponding rigid motion parameters in the rigid motion components of the hull at that time. The rigid motion parameters are converted into three-dimensional rotation transformation operators and three-dimensional translation transformation operators. The inverse transformation of the three-dimensional rotation transformation operator and the inverse transformation of the three-dimensional translation transformation operator are sequentially applied to the spatial sensing data in each data frame to generate multi-source spatial sensing data after decoupling of hull motion.

3. The method according to claim 1, characterized in that, Feature extraction is performed on the decoupled multi-source spatial sensing data to establish spatiotemporal correspondences between different sensing data sources, resulting in an aligned multimodal feature set, including: The original modal features of each sensing data source are extracted from the decoupled multi-source spatial sensing data. The original modal features characterize the spatial distribution features and temporal evolution features of the target object. Obtain the timestamps, spatial coordinate system parameters, sampling frequency, and data format attributes of each sensing data source, and analyze the temporal and spatial heterogeneity of different sensing data sources. To address time domain heterogeneity, the sampling times of each sensing data source are mapped to a unified time base, and a time synchronization relationship across data sources is established through timestamp interpolation; to address spatial domain heterogeneity, the spatial coordinates of each sensing data source are uniformly transformed to the ship's coordinate system, and a spatial alignment relationship across data sources is established. Based on the temporal synchronization relationship and spatial alignment relationship, cross-data source features that are within the same time window and point to the same spatial region are identified, and different modal features that meet the spatiotemporal association conditions are associated and bound to generate an aligned multimodal feature set.

4. The method according to claim 1, characterized in that, Based on the multimodal feature set, the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source are identified, and the confidence weight of each sensing data source is calculated, including: Based on the multimodal feature set, the spatial distribution features and temporal evolution features of each sensing data source representing the target object are extracted. Distance observation values ​​are extracted from the spatial distribution features. The shortest spatial connection path length is determined by combining the connectivity topology of the cabin structure, and the theoretical shortest propagation delay is calculated. The signal arrival time sequence is extracted from the time-series evolution characteristics, and the signal time of each sensing data source is mapped to a unified benchmark. When the actual propagation delay exceeds the propagation delay tolerance of the theoretical shortest propagation delay, an abnormal propagation path identifier is generated. Extract the time series of the received signal strength, analyze the rate of change of signal strength attenuation over time, calculate the theoretical signal strength attenuation value based on the material properties on the shortest spatial connection path, and mark the actual attenuation value as a signal strength attenuation indicator when it exceeds the attenuation tolerance of the theoretical attenuation value. The frequency of occurrence of the propagation path anomaly marker and the signal strength attenuation marker within a preset time window is statistically analyzed, converted into anomaly degree quantification values, and the confidence weight of each sensing data source is calculated.

5. The method according to claim 1, characterized in that, Based on the multimodal feature set, candidate localization solutions from each sensing data source are generated. According to the spatial connectivity constraints of the cabin structure data, physically inaccessible candidate localization solutions are eliminated. These solutions are then weighted and fused using the confidence weights to obtain the fused localization result of the target object, including: Based on the multimodal feature set, the spatial distribution features and temporal evolution features of each sensing data source are extracted, the spatial distribution features of each sensing data source are converted into spatial position estimates in the ship coordinate system, and the search space range is determined. Combined with the motion parameters in the temporal evolution features, candidate localization solutions for each sensing data source are generated in the search space. Based on the cabin structure data, the spatial connectivity topology of the cabin is extracted, a cabin spatial connectivity graph is constructed, the spatial coordinates of each candidate positioning solution are mapped to the nodes of the cabin spatial connectivity graph, the spatial unit node to which each candidate positioning solution belongs is determined, and candidate positioning solutions belonging to physically unreachable isolated nodes are eliminated to obtain valid candidate positioning solutions. Within a continuous time window, the spatiotemporal trajectory of valid candidate localization solutions is extracted, and the spatial displacement at adjacent time points is calculated. When the displacement exceeds the motion range predicted based on temporal evolution features, it is marked as a motion anomaly candidate localization solution. Based on the confidence weight, the valid candidate localization solutions not marked as motion anomalies are weighted and fused to obtain the fused localization result of the target object.

6. The method according to claim 5, characterized in that, The spatial distribution characteristics of each sensing data source are converted into spatial position estimates in the ship's coordinate system, and the search space range is determined. Combining the motion parameters in the temporal evolution characteristics, candidate localization solutions for each sensing data source are generated within the search space, including: The observation values ​​of each sensing data source are extracted from the spatial distribution characteristics. Based on the measurement model of each sensing data source, the observation covariance of the observation values ​​is calculated and mapped to the scale parameter of the search space. The observation values ​​are mapped to the ship coordinate system to generate the spatial position estimate of each sensing data source. Using the geometric center of the spatial position estimate of each sensing data source as the reference point, the boundary coordinates of the search space are determined according to the scale parameter to construct the candidate position search space. Motion parameters are extracted from the temporal evolution features. The motion parameters include velocity vectors and acceleration vectors. Based on the velocity vectors and acceleration vectors, the position offset of each sensing data source at the next moment is predicted. The position offset is superimposed on the spatial position estimate of each sensing data source to obtain the predicted position of each sensing data source. The predicted position is subjected to spatial constraint determination of the candidate position search space. When the predicted position is within the boundary coordinate range of the candidate position search space, the predicted position is used as a candidate positioning solution. When the predicted position exceeds the boundary coordinate range, the predicted position is projected onto the boundary, and the projected position is used as a candidate positioning solution.

7. The method according to claim 1, characterized in that, The fused positioning results are matched with the topology of the decks inside the cabin. Vertical floor constraints are used to distinguish target objects on different deck levels, generating a 3D positioning result containing deck level identifiers, including: Extract the vertical coordinate range and inter-layer spacing information of each deck layer from the cabin structure data, construct a vertical dimension constraint model of the deck layer, obtain the three-dimensional spatial coordinates of the target object in the fusion positioning result, compare the vertical component of the three-dimensional spatial coordinates with the vertical dimension constraint model, and determine the candidate deck layer set to which the target object belongs. Based on the candidate deck layer set, extract the planar connectivity region and spatial accessibility information of the corresponding deck layer from the topology of the cabin deck, and establish the horizontal dimension topological constraints of the candidate deck layer. The horizontal coordinates of the target object in the fusion positioning result are projected onto the planar connected region corresponding to the candidate deck layer set. The spatial distance deviation between the projection point and the boundary of each connected region is calculated. Based on the spatial distance deviation and the horizontal dimension topological constraint, the target deck layer that matches the horizontal coordinates of the target object is selected from the candidate deck layer set. The hierarchical identifier of the target deck layer is associated and bound with the three-dimensional spatial coordinates in the fused positioning result to generate a three-dimensional positioning result containing the deck layer identifier.

8. A high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning system for passenger roll-on / roll-off ship cabins, used to implement the method as described in any one of claims 1-7, characterized in that, include: The first unit is used to acquire cabin structure data of passenger roll-on / roll-off ships, six-degree-of-freedom motion parameters of the hull, and multi-source spatial perception data of the target object; The second unit is used to decompose the rigid motion components of the hull based on the six degrees of freedom motion parameters of the hull, and to synchronously compensate the multi-source spatial sensing data to eliminate the coupling effect of the hull motion and obtain the multi-source spatial sensing data after the hull motion is decoupled. The third unit is used to extract features from the decoupled multi-source spatial sensing data, establish the spatiotemporal correspondence between different sensing data sources, and obtain an aligned multimodal feature set. The fourth unit is used to identify the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source based on the multimodal feature set, and to calculate the confidence weight of each sensing data source. The fifth unit is used to generate candidate localization solutions for each sensing data source based on the multimodal feature set, eliminate physically inaccessible candidate localization solutions according to the spatial connectivity constraints of the cabin structure data, and perform weighted fusion based on the confidence weight to obtain the fused localization result of the target object. The sixth unit is used to match the fused positioning results with the topology of the deck inside the cabin, use the vertical dimension floor constraints to distinguish target objects of different deck layers, and generate a three-dimensional positioning result containing deck layer identifiers.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images