Multi-source heterogeneous sensing fusion positioning method for high-dynamic three-dimensional scene in passenger roll cabin

Abstract

The invention provides a multi-source heterogeneous sensing fusion positioning method for a high-dynamic three-dimensional scene in a passenger roll cabin, and relates to the technical field of cabin positioning, and the method comprises the steps: obtaining cabin structure data, six-degree-of-freedom motion parameters of a ship body, and multi-source sensing data; carrying out hull motion decoupling processing on the sensing data; establishing a multi-modal feature corresponding relation; calculating the confidence coefficient weight of each sensing source; according to cabin structure constraints, rejecting physically unreachable positioning solutions and carrying out weighted fusion; and generating a three-dimensional positioning result in combination with a deck topological structure. The ship dynamic environment interference is effectively overcome, and the multi-source heterogeneous sensing fusion positioning precision and reliability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ship cabin positioning, and in particular to a high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning method in a roll-on / roll-off passenger ship cabin. BACKGROUND

[0002] With the continuous development of marine tourism and passenger transportation, the safety and service quality of roll-on / roll-off passenger ships as important water transportation tools are increasingly concerned. The cabin of a roll-on / roll-off passenger ship usually has a complex structure, including multiple decks and various functional areas, and is crowded with people and frequent activities. In order to improve the level of passenger safety protection, optimize emergency response capabilities, and improve ship operation management efficiency, accurate positioning of target objects (such as personnel and equipment) in the cabin is needed. In the cabin environment of a roll-on / roll-off passenger ship, due to factors such as metal structure shielding, electromagnetic interference, personnel flow, and the movement of the ship itself, a single perception technology cannot provide stable and reliable positioning services. At present, indoor positioning technologies such as WiFi, Bluetooth, and UWB have been widely used in land buildings, but there are still some problems in the special scenario of a roll-on / roll-off passenger ship. SUMMARY

[0003] The embodiments of the present application provide a high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning method in a roll-on / roll-off passenger ship cabin, which can solve the problems in the prior art.

[0004] In a first aspect, the embodiments of the present application provide a high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning method in a roll-on / roll-off passenger ship cabin, comprising: obtaining cabin structure data of the roll-on / roll-off passenger ship cabin, six-degree-of-freedom motion parameters of the ship body, and multi-source spatial perception data of a target object; According to the six-degree-of-freedom motion parameters of the ship body, the rigid motion component of the ship body is decomposed, and the multi-source spatial perception data is compensated synchronously to eliminate the coupling effect of the ship body motion, and the multi-source spatial perception data after decoupling of the ship body motion is obtained; performing feature extraction on the decoupled multi-source spatial perception data, establishing a spatio-temporal correspondence relationship between different perception data sources, and obtaining an aligned multi-modal feature set; Based on the multi-modal feature set, the propagation path anomaly identifier and the signal strength attenuation identifier of each perception data source are identified, and the confidence weight of each perception data source is calculated; Based on the multi-modal feature set, candidate positioning solutions of each perception data source are generated, physically unattainable candidate positioning solutions are eliminated according to the spatial connectivity constraint of the cabin structure data, and the fusion positioning result of the target object is obtained by weighted fusion combined with the confidence weight; The fusion positioning result is matched with the topological structure of the cabin deck, the target objects on different deck layers are distinguished by using the floor constraint in the vertical dimension, and a three-dimensional positioning result containing deck layer identifiers is generated.

[0005] According to the six-degree-of-freedom motion parameters of the ship body, a ship body rigid motion component is decomposed, and the multi-source spatial perception data is synchronously compensated to eliminate the coupling effect of the ship body motion, so as to obtain the multi-source spatial perception data after decoupling of the ship body motion, comprising: The roll angle time sequence, the pitch angle time sequence and the yaw angle time sequence in the six-degree-of-freedom motion parameters of the ship body are respectively subjected to frequency domain transformation, and the angle frequency domain components in the angle time sequence within the inherent rolling frequency range of the ship body are extracted. The angle frequency domain components are subjected to inverse Fourier transform to obtain the ship body rigid angle component; According to the frequency characteristics of the ship body rigid angle component, displacement frequency domain components of the same frequency are identified from the displacement time sequence in the six-degree-of-freedom motion parameters, and the ship body rigid displacement component is obtained by inverse Fourier transform. In combination with the ship body rigid angle component, the ship body rigid motion component is obtained; The acquisition time of each data frame in the multi-source spatial perception data is extracted, and the rigid motion parameters at the corresponding time in the ship body rigid motion component are searched; The rigid motion parameters are converted into a three-dimensional rotation transformation operator and a three-dimensional translation transformation operator, and the inverse transformation of the three-dimensional rotation transformation operator and the inverse transformation of the three-dimensional translation transformation operator are sequentially applied to the spatial perception data in each data frame to generate the multi-source spatial perception data after decoupling of the ship body motion.

[0006] The decoupled multi-source spatial perception data is subjected to feature extraction, and a spatio-temporal correspondence relationship between different perception data sources is established to obtain an aligned multi-modal feature set, comprising: Original modal features of each perception data source are extracted from the decoupled multi-source spatial perception data, and the original modal features represent the spatial distribution characteristics and the time sequence evolution characteristics of the target object; The time stamp, spatial coordinate system parameter, sampling frequency and data format attribute of each perception data source are obtained, and the time domain heterogeneity and the spatial domain heterogeneity of different perception data sources are analyzed; For the time domain heterogeneity, the sampling time of each perception data source is mapped to a unified time reference, and the time sequence synchronization relationship across data sources is established through time stamp interpolation. For the spatial domain heterogeneity, the spatial coordinates of each perception data source are uniformly converted to the ship body coordinate system, and the spatial alignment relationship across data sources is established; Based on the time sequence synchronization relationship and the spatial alignment relationship, the cross-data-source features in the same time window and pointing to the same spatial region are identified, and the different modal features satisfying the spatio-temporal correlation condition are correlated and bound to generate the aligned multi-modal feature set.

[0007] Based on the multi-modal feature set, a propagation path anomaly identifier and a signal strength attenuation identifier of each perception data source are identified, and a confidence weight of each perception data source is calculated, including: Based on the spatial distribution features and the time evolution features of each perception data source in the multi-modal feature set, a distance observation value is extracted from the spatial distribution features, a shortest spatially connected path length is determined based on the connectivity topology of the cabin structure, and a theoretical shortest propagation time delay is calculated; A signal arrival time sequence is extracted from the time evolution features, and the signal time of each perception data source is mapped to a unified reference. When the actual propagation time delay exceeds the propagation time delay tolerance of the theoretical shortest propagation time delay, a propagation path anomaly identifier is generated; A time sequence of received signal strength is extracted, the signal strength attenuation rate over time is analyzed, and a theoretical signal strength attenuation value is calculated based on the material properties on the shortest spatially connected path. When the actual attenuation value exceeds the attenuation tolerance of the theoretical attenuation value, it is marked as a signal strength attenuation identifier; The occurrence frequencies of the propagation path anomaly identifier and the signal strength attenuation identifier within a preset time window are counted, converted into an abnormality degree quantization value, and the confidence weight of each perception data source is calculated.

[0008] Based on the multi-modal feature set, candidate positioning solutions of each perception data source are generated, physically unreachable candidate positioning solutions are eliminated based on the spatial connectivity constraints of the cabin structure data, and the fusion positioning result of the target object is obtained by weighted fusion combined with the confidence weight, including: Based on the multi-modal feature set, spatial distribution features and time evolution features of each perception data source are extracted, the spatial distribution features of each perception data source are converted into spatial position estimation values in the ship body coordinate system, and the search space range is determined. Combined with the motion parameters in the time evolution features, candidate positioning solutions of each perception data source are generated within the search space; According to the cabin structure data, the spatial connectivity topology inside the cabin is extracted, and a cabin spatial connectivity graph is constructed. The spatial coordinates of each candidate positioning solution are mapped to the nodes of the cabin spatial connectivity graph, the spatial unit nodes to which each candidate positioning solution belongs are determined, and candidate positioning solutions belonging to physically unreachable isolated nodes are eliminated to obtain valid candidate positioning solutions; The spatiotemporal trajectory of the valid candidate positioning solutions is extracted within a continuous time window, and the spatial displacement amount at adjacent time instants is calculated. When the displacement amount exceeds the motion range predicted based on the time evolution features, it is marked as a motion abnormal candidate positioning solution. According to the confidence weight, the valid candidate positioning solutions that are not marked as motion abnormal are weighted and fused to obtain the fusion positioning result of the target object.

[0009] The spatial distribution characteristics of each perception data source are converted into spatial position estimation values in the ship body coordinate system, and the search space range is determined. In combination with the motion parameters in the time sequence evolution characteristics, candidate positioning solutions of each perception data source are generated in the search space, including: The observation values of each perception data source are extracted from the spatial distribution characteristics, the observation covariance of the observation values is calculated according to the measurement model of each perception data source and is mapped into the scale parameters of the search space, the observation values are mapped to the ship body coordinate system to generate the spatial position estimation values of each perception data source, and the geometric center of the spatial position estimation values of each perception data source is taken as the reference point. According to the scale parameters, the boundary coordinates of the search space are determined, and the candidate position search space is constructed; The motion parameters are extracted from the time sequence evolution characteristics, the motion parameters include velocity vectors and acceleration vectors, the position offset of each perception data source at the next time is predicted according to the velocity vectors and the acceleration vectors, and the position offset is superimposed on the spatial position estimation values of each perception data source to obtain the predicted position of each perception data source; The predicted position is subjected to the spatial constraint determination of the candidate position search space. When the predicted position is located within the boundary coordinate range of the candidate position search space, the predicted position is taken as the candidate positioning solution. When the predicted position exceeds the boundary coordinate range, the boundary projection is performed on the predicted position, and the projected position is taken as the candidate positioning solution.

[0010] The fusion positioning result is matched with the topological structure of the cabin deck, the target object of different deck layers is distinguished by using the vertical dimension constraint, a three-dimensional positioning result containing deck layer identification is generated, including: The vertical coordinate range and interlayer spacing information of each deck layer in the cabin structure data are extracted, a vertical dimension constraint model of the deck layer is constructed, the three-dimensional spatial coordinates of the target object in the fusion positioning result are obtained, the vertical component of the three-dimensional spatial coordinates is compared with the vertical dimension constraint model, and a candidate deck layer set to which the target object belongs is determined; According to the candidate deck layer set, the planar connected region and spatial accessibility information of the corresponding deck layer are extracted from the topological structure of the cabin deck to establish the horizontal dimension topological constraint of the candidate deck layer; The horizontal coordinates of the target object in the fusion positioning result are projected to the planar connected region corresponding to the candidate deck layer set, the spatial distance deviation between the projection point and the boundary of each connected region is calculated, and according to the spatial distance deviation and the horizontal dimension topological constraint, the target deck layer matched with the horizontal coordinates of the target object is screened out from the candidate deck layer set; The level identification of the target deck layer is associated and bound with the three-dimensional space coordinates in the fusion positioning result, and a three-dimensional positioning result containing the deck layer identification is generated.

[0011] In a second aspect of the embodiment of the application, a high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning system for a passenger-rolling ship cabin is provided, and the system comprises: A first unit is configured to acquire cabin structure data of the passenger-rolling ship cabin, six-degree-of-freedom motion parameters of a ship body, and multi-source spatial perception data of a target object. A second unit is configured to decompose a rigid motion component of the ship body according to the six-degree-of-freedom motion parameters of the ship body, and perform synchronous compensation on the multi-source spatial perception data to eliminate the coupling effect of the ship body motion, so as to obtain the multi-source spatial perception data after the ship body motion is decoupled. A third unit is configured to perform feature extraction on the decoupled multi-source spatial perception data, establish a spatio-temporal correspondence relationship between different perception data sources, and obtain an aligned multi-modal feature set. A fourth unit is configured to identify a propagation path abnormality identifier and a signal strength attenuation identifier of each perception data source based on the multi-modal feature set, and calculate a confidence weight of each perception data source. A fifth unit is configured to generate a candidate positioning solution of each perception data source based on the multi-modal feature set, eliminate physically unattainable candidate positioning solutions according to a spatial connectivity constraint of the cabin structure data, combine the confidence weight for weighted fusion, and obtain a fusion positioning result of the target object. A sixth unit is configured to match the fusion positioning result with a topological structure of a cabin deck, distinguish target objects of different deck layers by using a vertical dimension floor constraint, and generate a three-dimensional positioning result containing a deck layer identification.

[0012] In a third aspect of the embodiment of the application, An electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0013] In a fourth aspect of the embodiment of the application, A computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0014] The application has the following beneficial effects: through the six-degree-of-freedom motion parameter decomposition and compensation mechanism, the interference of high-dynamic ship body motion on positioning accuracy is effectively eliminated, and the perception data remains stable in a high-dynamic ship environment.

[0015] The feature extraction and spatio-temporal correspondence relationship establishment method using multi-source heterogeneous perception data realizes effective alignment and fusion of different perception technology data, and overcomes the limitations of single perception technology in a complex ship environment. The propagation path anomaly identification and signal strength attenuation identification recognition mechanism is introduced, and the dynamic calculation of the confidence weight is combined, which significantly improves the positioning robustness in a complex electromagnetic environment and a multi-obstacle environment.

[0016] Based on the spatial connectivity constraint of the cabin structure data, the physically unreachable candidate positioning solution is effectively eliminated, and the fuzzy positioning problem of the traditional positioning method in the multi-deck structure is solved. By distinguishing different deck layers through the vertical dimension floor constraint, the three-dimensional positioning in a true sense is realized, the deck layer where the target object is located is accurately identified, and the practicality of the positioning result in the three-dimensional ship environment is improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning method in a passenger-rolling ship cabin according to an embodiment of the present application is shown in Figure 2 A flowchart of a method for generating multi-source spatial perception data after decoupling ship body motion according to an embodiment of the present application is shown in DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0019] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0020] Figure 1 A flowchart of a high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning method in a passenger-rolling ship cabin according to an embodiment of the present application is shown in Figure 1 As shown in the figure, the method comprises: obtaining cabin structure data of a passenger-rolling ship cabin, six-degree-of-freedom motion parameters of a ship body, and multi-source spatial perception data of a target object; According to the six-degree-of-freedom motion parameters of the ship body, the rigid motion component of the ship body is decomposed, and the multi-source spatial perception data is compensated synchronously to eliminate the coupling effect of the ship body motion, and multi-source spatial perception data after decoupling of the ship body motion is obtained; perform feature extraction on the decoupled multi-source spatial perception data, establish a spatio-temporal correspondence relationship between different perception data sources, and obtain an aligned multi-modal feature set; Based on the multi-modal feature set, identify the propagation path anomaly identifier and signal strength attenuation identifier of each perception data source, and calculate the confidence weight of each perception data source; Based on the multi-modal feature set, generate candidate positioning solutions of each perception data source, eliminate physically inaccessible candidate positioning solutions according to the spatial connectivity constraint of the cabin structure data, combine the confidence weight for weighted fusion, and obtain a fusion positioning result of the target object; Match the fusion positioning result with the topological structure of the cabin deck, use the floor constraint in the vertical dimension to distinguish target objects on different deck layers, and generate a three-dimensional positioning result containing deck layer identifiers.

[0021] Figure 2 A flowchart of a method for generating multi-source spatial perception data decoupled from ship motion is generated for the embodiment of the application. In an optional implementation, the rigid motion component of the ship is decomposed according to the six-degree-of-freedom motion parameters of the ship, and the multi-source spatial perception data is compensated synchronously to eliminate the coupling effect of the ship motion, thereby obtaining multi-source spatial perception data decoupled from the ship motion, including: Perform frequency domain transformation on the roll angle time series, pitch angle time series and yaw angle time series in the six-degree-of-freedom motion parameters of the ship, extract the angle frequency components in the angle time series within the natural rolling frequency range of the ship, and perform inverse Fourier transform on the angle frequency components to obtain the rigid angle component of the ship; According to the frequency characteristics of the rigid angle component of the ship, identify the displacement frequency components of the same frequency from the displacement time series in the six-degree-of-freedom motion parameters, obtain the rigid displacement component of the ship by inverse Fourier transform, and combine the rigid angle component of the ship to obtain the rigid motion component of the ship; Extract the acquisition time of each data frame in the multi-source spatial perception data, and find the rigid motion parameters at the corresponding time in the rigid motion component; Convert the rigid motion parameters into a three-dimensional rotation transformation operator and a three-dimensional translation transformation operator, and sequentially apply inverse transformation of the three-dimensional rotation transformation operator and inverse transformation of the three-dimensional translation transformation operator to the spatial perception data in each data frame to generate multi-source spatial perception data decoupled from the ship motion.

[0022] During the navigation of a ship, the six-degree-of-freedom motion parameters of the ship body need to be collected, including the roll angle time series α(t), the pitch angle time series β(t), the yaw angle time series γ(t), and the corresponding longitudinal displacement time series x(t), the lateral displacement time series y(t), and the vertical displacement time series z(t). These parameters can be continuously recorded by a ship-borne inertial measurement unit.

[0023] The three groups of angle time series are respectively subjected to frequency domain transformation, and the angle frequency domain components in the angle time series within the inherent rolling frequency range of the ship body are extracted. Specifically, the fast Fourier transform (FFT) is performed on α(t), β(t), and γ(t) respectively to obtain the frequency domain representations α(f), β(f), and γ(f). The inherent rolling frequency of the ship body is usually between 0.05 Hz and 0.5 Hz, and varies according to the type and size of the ship. Therefore, by extracting the components within this frequency range from the frequency domain representations, the frequency domain features of the rigid motion of the ship body can be obtained.

[0024] The extracted angle frequency domain components are converted back to the time domain by inverse Fourier transform to obtain the rigid angle components α_rigid(t), β_rigid(t), and γ_rigid(t) of the ship body. These components represent the rolling motion of the ship body as a rigid body under the action of sea waves, and eliminate the influence of high-frequency vibration and noise.

[0025] According to the frequency characteristics of the rigid angle components of the ship body, the displacement frequency domain components of the same frequency are identified from the displacement time series in the six-degree-of-freedom motion parameters, the fast Fourier transform is performed on the displacement time series x(t), y(t), and z(t) to obtain the frequency domain representations x(f), y(f), and z(f), the displacement frequency domain components are extracted within the same frequency interval as the rigid angle components, and the inverse Fourier transform is performed to obtain the rigid displacement components x_rigid(t), y_rigid(t), and z_rigid(t) of the ship body. In combination with the rigid angle components of the ship body, a complete description of the rigid motion components of the ship body is formed.

[0026] In the process of acquiring multi-source spatial perception data, each data frame has an accurate time stamp, the acquisition time t_frame of each data frame in the multi-source spatial perception data is extracted, the rigid motion parameters at the corresponding time are searched in the rigid motion components of the ship body, and if the time stamp t_frame and the sampling time of the rigid motion parameters are not completely consistent, the linear interpolation method can be used to calculate the rigid motion parameters at that time, i.e. α_rigid(t_frame), β_rigid(t_frame), γ_rigid(t_frame), x_rigid(t_frame), y_rigid(t_frame), and z_rigid(t_frame).

[0027] The rigid motion parameters are converted into a three-dimensional rotation transformation operator and a three-dimensional translation transformation operator. The three-dimensional rotation transformation operator can be represented by a rotation matrix, which is constructed according to the Euler angles α_rigid(t_frame), β_rigid(t_frame), and γ_rigid(t_frame). The rotation matrix R can be obtained by a composite matrix of rotations around the z-axis, y-axis, and x-axis in sequence. The three-dimensional translation transformation operator is represented by a displacement vector T = [x_rigid(t_frame), y_rigid(t_frame), z_rigid(t_frame)].

[0028] For each frame of data in the multi-source spatial perception data, which contains a three-dimensional point cloud or spatial coordinate point P, the inverse transformation of the three-dimensional rotation transformation operator and the inverse transformation of the three-dimensional translation transformation operator are sequentially applied to the three-dimensional point cloud or spatial coordinate point P. The specific calculation is P' = R (-1) · (P-T), where R (-1) is the inverse matrix of the rotation matrix, which is actually equal to the transpose of the rotation matrix. Each spatial point is transformed into the ship reference coordinate system, eliminating the influence of ship motion.

[0029] In practical applications, for example, when a marine survey ship carries a multi-beam sonar system for seabed topography mapping, the ship's sway motion under the action of waves will cause the distortion and overlap of sonar scan lines. By the above method, the scan line data can be converted from the acquisition coordinate system to the ship reference coordinate system, and the geometric distortion caused by ship motion can be corrected, so that a more accurate seabed topography map can be obtained.

[0030] Similarly, for the three-dimensional point cloud data of port facilities collected by the ship-borne laser radar system, the above decoupling method can be applied to eliminate the influence of ship sway on the geometric shape of the point cloud, ensuring the spatial accuracy of the data. For the image sequence obtained by the ship-borne camera system, the accuracy of image stitching and target recognition can be improved by compensating for ship motion.

[0031] Through the above steps, the multi-source spatial perception data generation process after ship motion decoupling is completed, providing high-quality input for subsequent data analysis and processing. The data after ship motion decoupling can more accurately reflect the real spatial characteristics of the environment, improving the accuracy of ocean exploration and environmental perception.

[0032] In an optional implementation, feature extraction is performed on the decoupled multi-source spatial perception data to establish a spatio-temporal correspondence relationship between different perception data sources, and an aligned multi-modal feature set is obtained, including: Original modal features of each perception data source are extracted from the decoupled multi-source spatial perception data. The original modal features represent the spatial distribution characteristics and time sequence evolution characteristics of the target object. Obtain the timestamp, spatial coordinate system parameter, sampling frequency and data format attribute of each perception data source, analyze the time domain heterogeneity and spatial domain heterogeneity of different perception data sources; For time domain heterogeneity, map the sampling time of each perception data source to a unified time reference, and establish a cross-data-source time synchronization relationship through timestamp interpolation; for spatial domain heterogeneity, unify the spatial coordinates of each perception data source to the ship body coordinate system, and establish a cross-data-source spatial alignment relationship; Based on the time synchronization relationship and spatial alignment relationship, identify the cross-data-source features in the same time window and pointing to the same spatial region, and bind the different modal features that meet the spatio-temporal correlation condition to generate an aligned multi-modal feature set.

[0033] In this embodiment, in the cruise ship perception system, the radar device is usually installed on the upper deck of the ship to ensure full coverage; the photoelectric device is distributed in the key positions of each deck of the ship, focusing on monitoring the passageway, hatch and cargo area; the sonar device is installed at the bottom of the ship for underwater environment perception, and the automatic identification system antenna is installed at the top of the ship mast to ensure signal reception quality. These devices together constitute a multi-source perception network of the ship, and obtain decoupled multi-source spatial perception data for original modal feature extraction. For radar data, the reflection intensity feature, echo width feature and Doppler velocity feature of the target echo signal are extracted through the range-azimuth detection algorithm. These features represent the electromagnetic reflection characteristics and relative motion state of the target object. For photoelectric data, the shape contour feature, texture feature and color feature of the target region are extracted by applying image segmentation algorithm. These features represent the visual appearance characteristics of the target object. For sonar data, acoustic features are extracted through acoustic spectrum analysis, including frequency distribution feature, energy feature and sound source direction feature. These features represent the acoustic radiation characteristics of the target object. For automatic identification system data, ship identity, navigation state and static information features are extracted.

[0034] The extracted original modal features jointly constitute the representation basis of the spatial distribution features and the time evolution features of the target object. In terms of the spatial distribution features, the reflection intensity distribution and the range-azimuth data provided by the radar constitute the spatial profile of the target; the shape and color information provided by the photoelectric data enhances the spatial details; the sonar data supplements the underwater spatial information; and the automatic identification system data provides a relative spatial positioning reference. By fusing these spatial distribution features, a full-spectrum spatial perception of the target object in the cruise ship environment is formed. In terms of the time evolution features, the continuously collected radar Doppler velocity features depict the motion trajectory of the target; the photoelectric data sequence reflects the time-varying characteristics of the target appearance; the time sequence change of the sonar data reveals the behavior mode of the underwater target; and the dynamic information update of the automatic identification system reflects the sailing dynamics of the surrounding ships. These time evolution features enable the positioning system to predict the future position of the target and adapt to the high dynamic environment of the cruise ship.

[0035] The timestamps, spatial coordinate system parameters, sampling frequencies, and data format attributes of each perception data source are obtained for analyzing the time domain heterogeneity and spatial domain heterogeneity of different perception data sources. For radar data, the scanning period (usually 1-4 seconds / week), sampling timestamp, installation position of the radar antenna relative to the ship body, and radar coordinate system parameters are obtained. For photoelectric equipment, the imaging frame rate (usually 25-30 frames / second), image acquisition timestamp, installation position of the photoelectric platform, and rotation angle parameters are obtained. For sonar equipment, the sound wave transmission and reception period, signal processing timestamp, installation position of the sonar array relative to the ship body, and sonar coordinate system parameters are obtained. For the automatic identification system, the data update period (usually 2-10 seconds) and reception timestamp are obtained. By analyzing these parameters, it is found that different data sources have time domain heterogeneity (inconsistent sampling frequencies and different time references) and spatial domain heterogeneity (different installation positions of each sensor and inconsistent coordinate system definitions).

[0036] For time domain heterogeneity, the sampling time of each perception data source is mapped to a unified time reference, and the time sequence synchronization relationship across data sources is established through timestamp interpolation. The GPS time of the ship navigation system is selected as the unified time reference, and the timestamps of all data sources are converted to this time reference. For data with time delay, the time offset is calculated and corrected according to the hardware processing delay and signal transmission delay. For data sources with inconsistent sampling frequencies, a time window segmentation processing method is used, and an appropriate time window length (such as 1 second) is selected. The sampling points of each data source within the window are mapped to the unified time of the window midpoint through linear interpolation or spline interpolation. For periodically sampled data sources (such as radar rotation scanning), the observation data estimate value at any time is calculated based on its periodic characteristics and scanning direction. Through these processing, the different perception data sources are uniformly aligned in the time dimension.

[0037] For spatial domain heterogeneity, the spatial coordinates of each perception data source are uniformly converted to the ship body coordinate system, the spatial alignment relationship across data sources is established, the ship body coordinate system is defined, and the center of gravity or a specific reference point of the ship is usually selected as the coordinate origin. The positive direction of the bow is the positive direction of the x-axis, the positive direction of the right side is the positive direction of the y-axis, and the vertical upward direction is the positive direction of the z-axis. According to the installation position and installation attitude of each sensor, the coordinate transformation matrix from the sensor local coordinate system to the ship body coordinate system is established. For radar data, the target position represented in the polar coordinate system (distance-azimuth angle) is converted to the Cartesian coordinate system, and then converted to the ship body coordinate system through the coordinate transformation matrix. For optical data, the target position in the image pixel coordinate system is converted to the ship body coordinate system through the camera intrinsic and extrinsic parameter matrices. For sonar data, the sonar bearing-elevation and ranging information is converted to the three-dimensional position in the ship body coordinate system. Through these coordinate transformations, the different perception data sources are uniformly aligned in the spatial dimension.

[0038] Based on the timing synchronization relationship and the spatial alignment relationship, cross-data-source features pointing to the same spatial region within the same time window are identified, different modal features satisfying the spatio-temporal correlation condition are associated and bound, an aligned multi-modal feature set is generated, and in specific implementation, the judgment criteria for spatio-temporal correlation are defined: in the time dimension, if the sampling time difference of two data sources is less than a preset threshold (such as 0.5 seconds), it is considered to satisfy the time correlation condition; in the spatial dimension, if the Euclidean distance of the target spatial position pointed to by two data sources is less than a preset threshold (such as 10 meters), it is considered to satisfy the spatial correlation condition. For different modal features satisfying the spatio-temporal correlation condition, a unique correlation identifier is generated, and these features are bound into a multi-modal feature vector. For one-to-many or many-to-one association cases, nearest neighbor or probabilistic data association algorithms are used for optimized matching to generate an aligned multi-modal feature set. Each set element contains a combination of features from different perception data sources pointing to the same target object.

[0039] Through the above steps, feature extraction of decoupled multi-source spatial perception data and spatio-temporal alignment across data sources are realized, providing basic data support for subsequent multi-modal fusion perception.

[0040] In an optional implementation, based on the multi-modal feature set, the propagation path anomaly identifier and the signal strength attenuation identifier of each perception data source are identified, and the confidence weight of each perception data source is calculated, including: Based on the spatial distribution features and the time sequence evolution features of the target object represented by each perception data source in the multi-modal feature set, the distance observation value is extracted from the spatial distribution features, the shortest spatial connected path length is determined according to the connectivity topology of the cabin structure, and the theoretical shortest propagation time delay is calculated. Extract the signal arrival time sequence from the time evolution feature, map the signal time of each perception data source to a unified reference, and generate a propagation path anomaly identifier when the actual propagation delay exceeds the propagation delay tolerance of the theoretical shortest propagation delay. Extract the time sequence of received signal strength, analyze the decay rate of signal strength over time, and calculate the theoretical signal strength decay value based on the material properties on the shortest spatially connected path. When the actual decay value exceeds the decay tolerance of the theoretical decay value, it is marked as a signal strength decay identifier. Statistical analysis of the occurrence frequency of the propagation path anomaly identifier and the signal strength decay identifier within the preset time window, convert the abnormal degree quantitative value and calculate the confidence weight of each perception data source.

[0041] In this embodiment, the spatial distribution features and time evolution features of each perception data source representing the target object are extracted from the multi-modal feature set. When extracting spatial distribution features, focus on distance observation values representing the spatial position of the target object. For wireless signal systems, convert the received signal strength indicator value to a distance estimate value based on the correspondence between the signal strength indicator value and the signal source. For visual sensors, calculate the depth information of the target in the image by triangulation principle and convert it to actual distance value. The extracted distance observation values are stored in vector form, containing target ID, observation timestamp and distance value triplets.

[0042] When determining the length of the shortest spatially connected path based on the connectivity topology of the cabin structure, a three-dimensional topological graph model of the passenger-cargo ship cabin is established. This model divides the cabin into multiple interconnected nodes, with nodes representing cabins, corridors or key locations, and edges representing direct connectivity between nodes. The weight of the edge is set according to the actual physical distance, taking into account factors such as corridor width and obstacle distribution that affect connectivity. The shortest path between the signal source and the receiving point is searched on the topological graph through breadth-first search, and the length of the shortest spatially connected path is obtained. This path length reflects the theoretical shortest distance of signal propagation in the cabin, providing a reference for subsequent propagation delay analysis.

[0043] When calculating the theoretical shortest propagation delay, different types of signals are processed differently based on their propagation characteristics. Based on the shortest spatially connected path length and signal propagation speed (such as sound wave propagation speed of about 340 meters / second and electromagnetic wave propagation speed of about light speed), the theoretical shortest propagation delay is calculated. For example, if the shortest spatially connected path length is 15 meters, for a sound wave sensor, the theoretical shortest propagation delay is about 0.044 seconds.

[0044] The time sequence of signal arrival is extracted from the time evolution characteristics. Since different perception data sources adopt different time bases, the signal time of each perception data source needs to be mapped to a unified reference. The system global clock or a certain specific event can be selected as the reference point for time calibration. The actual propagation delay, i.e. the time difference from the signal being sent to being received by the perception data source, is calculated. When the actual propagation delay exceeds the propagation delay tolerance of the theoretical minimum propagation delay, an abnormal propagation path identifier is generated. The propagation delay tolerance can be set to 20% of the theoretical value, i.e. for the foregoing example, if the actual propagation delay exceeds 0.0528 seconds (0.044x1.2), it is marked as an abnormal propagation path.

[0045] At the same time, the time sequence of received signal strength is extracted, and the rate of change of signal strength over time is analyzed. According to the material properties on the shortest spatially connected path, such as the wall material and obstacle type through which the path passes, the theoretical signal strength attenuation value is calculated. For example, for a radio signal, in free space, the signal strength attenuates by about 6dB for every doubling of distance, and an additional 15-20dB is attenuated through a metal wall. When the actual attenuation value exceeds the attenuation tolerance of the theoretical attenuation value, it is marked as a signal strength attenuation identifier. The attenuation tolerance can be set to 30% of the theoretical attenuation value, or adjusted dynamically according to the complexity of the environment.

[0046] In actual application, a certain perception data source generates 100 data points within a 10-second sampling window. If 15 data points are detected to have an abnormal propagation path identifier and 12 data points have a signal strength attenuation identifier, the abnormality rate in this window is 15% and 12% respectively. The frequency of occurrence of the propagation path abnormality identifier and the signal strength attenuation identifier in the preset time window is counted. The time window can be set as a sliding window, such as the last 10 seconds or the last 100 sampling points. The abnormality frequency is converted into an abnormality degree quantization value. The sigmoid function can be used to map the abnormality frequency to the [0, 1] interval. For example, for the propagation path abnormality, the abnormality degree quantization value can be calculated as 1 / (1+exp(-(abnormality frequency-0.2)*10)). The signal strength attenuation abnormality degree quantization value can be calculated using a similar method.

[0047] Based on the abnormality degree quantization value, the confidence weight of each perception data source is calculated. The confidence weight is inversely proportional to the abnormality degree quantization value, which can be represented as 1 minus the weighted sum of the abnormality degree quantization value. The weighting coefficient can be adjusted according to the application scenario, such as a propagation path abnormality weight of 0.6 and a signal strength attenuation abnormality weight of 0.4. For example, if the propagation path abnormality degree quantization value of a certain perception data source is 0.3 and the signal strength attenuation abnormality degree quantization value is 0.2, then its confidence weight is 1-(0.3x0.6+0.2x0.4)=0.74.

[0048] In practical application scenarios, such as personnel positioning in a closed cabin, when the signal received by a certain perception data source (such as a WiFi signal strength sensor) exhibits obvious propagation path abnormalities, it is due to signal reflection or diffraction, at which time the weight of this data source in the fusion positioning should be reduced. Similarly, when the target thermal signal strength detected by an infrared thermal imaging sensor decays abnormally, it is due to the existence of unrecorded obstructions between the target and the sensor, and the credibility of the sensor data should be reduced accordingly.

[0049] The confidence weight calculated by the above method can be used for subsequent multi-source data fusion processing to achieve more accurate estimation of the position of the target object and improve the overall perception accuracy of the system.

[0050] In an optional implementation, based on the multi-modal feature set, candidate positioning solutions of each perception data source are generated, physically unattainable candidate positioning solutions are eliminated according to the spatial connectivity constraint of the cabin structure data, weighted fusion is performed in combination with the confidence weight to obtain a fusion positioning result of the target object, including: Based on the multi-modal feature set, spatial distribution features and time sequence evolution features of each perception data source are extracted, the spatial distribution features of each perception data source are converted into spatial position estimation values in the ship body coordinate system, and a search space range is determined, candidate positioning solutions of each perception data source are generated in the search space in combination with the motion parameters in the time sequence evolution features; According to the cabin structure data, the spatial connectivity topology inside the cabin is extracted, a cabin spatial connectivity graph is constructed, the spatial coordinates of each candidate positioning solution are mapped to the nodes of the cabin spatial connectivity graph, the spatial unit nodes to which each candidate positioning solution belongs are determined, candidate positioning solutions belonging to physically unattainable isolated nodes are eliminated, and valid candidate positioning solutions are obtained; The spatiotemporal trajectory of the valid candidate positioning solutions is extracted within a continuous time window, the spatial displacement amount of adjacent time instants is calculated, and when the displacement amount exceeds the motion range predicted based on the time sequence evolution features, it is marked as a motion abnormal candidate positioning solution; according to the confidence weight, the valid candidate positioning solutions that are not marked as motion abnormal are weighted and fused to obtain a fusion positioning result of the target object.

[0051] In this specific embodiment, based on the multimodal feature set, the spatial distribution features of each sensing data source are mapped to a unified ship coordinate system through coordinate system transformation. For example, for thermal imaging data, the two-dimensional position of the hotspot area is converted into a three-dimensional spatial position estimate using known installation position parameters and field of view information; for camera data, monocular or binocular visual positioning methods are used, combined with known camera intrinsic and extrinsic parameters, to convert the target position in the image plane into a three-dimensional position estimate in the ship coordinate system; for acoustic data, based on the spatial layout of the microphone array, the azimuth angle and estimated distance of the sound source are converted into a spatial position estimate in the ship coordinate system. Taking the current spatial position estimate as the center, and combining the motion parameters in the temporal evolution features, a three-dimensional search sphere or ellipsoid is set. Within this search space, candidate positioning solutions are generated for each sensing data source through methods such as particle filtering or Monte Carlo sampling. These candidate positioning solutions represent the positional distribution of the target object.

[0052] Candidate positioning solutions are selected based on the cabin structure data. The cabin structure data is processed to extract structural elements such as walls, passages, doors and windows inside the cabin. A topological structure representing the connectivity of the cabin space is constructed. Specifically, the cabin space is discretized into grid cells. Each passable grid cell is used as a node in the spatial connectivity graph. Connection edges are established between adjacent passable cells to form the cabin space connectivity graph.

[0053] For each candidate location solution, its spatial coordinates are mapped to the nearest spatial cell node to determine the spatial cell to which it belongs. Using a breadth-first search or depth-first search algorithm, it is checked whether the spatial cell has a connected path to the known starting point or reference point. If no connected path exists, the candidate location solution is determined to belong to a physically unreachable isolated node and should be eliminated. For example, if a candidate location solution is located in an area completely isolated from the current known location by a wall and cannot be reached through doors, windows, or other passages, then the candidate location solution is eliminated. In this way, all the remaining candidate location solutions are physically reachable and valid candidate location solutions.

[0054] To further improve positioning accuracy, the spatiotemporal trajectories of valid candidate positioning solutions are extracted within a continuous time window. Specifically, valid candidate positioning solutions at adjacent time points are connected to form a spatiotemporal trajectory sequence. The spatial displacement at adjacent time points is calculated and compared with the motion range predicted based on temporal evolution characteristics. When the actual displacement significantly exceeds the predicted motion range (e.g., more than twice the predicted range), the candidate positioning solution is marked as a motion anomaly candidate positioning solution.

[0055] For the valid candidate positioning solutions that are not marked as motion anomaly, according to the previously determined confidence weight of each perception data source, a weighted fusion is performed, and a weighted average position is calculated as an example of the fusion manner. Assuming that there are n valid candidate positioning solutions, the position coordinates of each positioning solution are P i The confidence weight of the corresponding perception data source is w i The target position P after fusion can be expressed as the weighted average of all valid candidate positioning solutions. In actual applications, the confidence weight is dynamically adjusted according to environmental conditions, for example, the weight of video data is increased in a light environment, and the weight of acoustic data is reduced in a noisy environment.

[0056] Through the above steps, the multi-modal perception data, cabin structure constraints and time evolution characteristics are combined to obtain an accurate and reliable target object fusion positioning result, effectively solving the precision and reliability problem of target positioning in a complex cabin environment, and enabling stable tracking and accurate positioning of the target under the condition of limited field of view and complex environment.

[0057] In an optional implementation, the spatial distribution characteristics of each perception data source are converted into a spatial position estimation value in the ship body coordinate system, and the search space range is determined, and the motion parameters in the time evolution characteristics are combined to generate candidate positioning solutions of each perception data source in the search space, including: extracting observation values of each perception data source from the spatial distribution characteristics, calculating observation covariances of the observation values according to the measurement model of each perception data source and mapping them into scale parameters of the search space, mapping the observation values to the ship body coordinate system to generate spatial position estimation values of each perception data source, taking the geometric center of the spatial position estimation values of each perception data source as a reference point, determining the boundary coordinates of the search space according to the scale parameters, and constructing a candidate position search space; Extracting motion parameters from the time evolution characteristics, the motion parameters including a velocity vector and an acceleration vector, predicting the position offset of each perception data source at the next time according to the velocity vector and the acceleration vector, and adding the position offset to the spatial position estimation value of each perception data source to obtain the predicted position of each perception data source. Performing spatial constraint judgment of the candidate position search space on the predicted position, when the predicted position is located within the boundary coordinate range of the candidate position search space, taking the predicted position as a candidate positioning solution, and when the predicted position exceeds the boundary coordinate range, performing boundary projection on the predicted position, and taking the projected position as a candidate positioning solution.

[0058] In the embodiment, observation values of each perception data source are extracted from the spatial distribution features, including target distance, azimuth and elevation information obtained by radar, photoelectric, sonar and other sensors. Taking the radar sensor as an example, the observation values include the slant range R, azimuth θ and elevation φ of the target. According to the installation position and attitude parameters of the sensor, a measurement model is established to convert the observation values into three-dimensional coordinates (x, y, z) in the ship coordinate system.

[0059] The calculation of the observation covariance of the observation value is a key step to ensure the position accuracy. The observation covariance reflects the measurement accuracy of each sensor, which is usually related to the measurement distance, angle and sensor error itself. For example, the radar distance measurement error σ R is proportional to the range, and the azimuth measurement error σ θ is related to the azimuth resolution. The observation covariance is mapped to the ship coordinate system through coordinate transformation to form an uncertainty ellipsoid of the position estimation, and the half-axis length is taken as the scale parameter of the search space.

[0060] After obtaining the spatial position estimation value of each perception data source, the geometric center is calculated as the reference point of the search space. The geometric center coordinates are the arithmetic mean of the position coordinates of each data source. Combined with the aforementioned scale parameter, the boundary coordinates of the search space are determined. The boundary range is usually set to be 2-3 times the scale parameter extended upwards and downwards in each dimension with the reference point as the center, forming a cuboid search space.

[0061] The time evolution feature provides target motion information, including velocity vector and acceleration vector. These parameters are obtained by difference calculation of continuous multiple frames of observation data. For newly obtained observation data, the position of each perception data source at the current time is predicted by combining the position estimation at the last time and the calculated motion parameters. The prediction process adopts the uniform acceleration motion model: new position = old position + velocity × time interval + 0.5 × acceleration × time interval squared.

[0062] Boundary constraint judgment of the search space for the predicted position is crucial. When the predicted position is within the boundary coordinate range, it is directly taken as the candidate positioning solution. When the predicted position exceeds the boundary, boundary projection processing is needed. The nearest point projection method is adopted to project the predicted position beyond the boundary to the boundary surface to obtain the candidate position that satisfies the spatial constraint.

[0063] Taking the multi-sensor cooperative tracking on the sea as an example, when the sonar and radar detect the target at the same time, the observation values of the two sensors are converted into the position estimation in the ship coordinate system. The sonar measurement accuracy is higher underwater, and the radar target detection is more accurate above the water surface. By calculating the observation covariance, the scale parameters of the search space are determined. The scale parameter of the sonar in the depth direction is smaller, and the scale parameter in the horizontal direction is larger. The radar is opposite. In combination with the historical motion trajectory of the target, the speed and acceleration information are extracted, and the position of the target at the current time is predicted. If the target is performing a rapid turning maneuver, the predicted position will exceed the boundary of the search space. At this time, the predicted position is projected onto the boundary of the search space through the boundary projection processing, and is used as the final candidate positioning solution.

[0064] In actual application, when there is a large difference between the position estimations of the multi-source data, a weighted average method can be used to determine the reference point of the search space. The weight is proportional to the observation accuracy of each sensor. By reasonably setting the range of the search space and scientifically processing the boundary constraint of the predicted position, the accuracy and stability of the multi-source perception data fusion positioning can be effectively improved.

[0065] In an optional implementation, the fusion positioning result is matched with the topology structure of the indoor deck, the target object of different deck layers is distinguished by using the floor constraint in the vertical dimension, and a three-dimensional positioning result containing a deck layer identifier is generated, including: Vertical coordinate ranges and interlayer interval information of each deck layer in the cabin structure data are extracted, a vertical dimension constraint model of the deck layer is constructed, three-dimensional space coordinates of the target object in the fusion positioning result are obtained, a vertical component of the three-dimensional space coordinates is compared with the vertical dimension constraint model, and a candidate deck layer set to which the target object belongs is determined; According to the candidate deck layer set, planar connected region and spatial accessibility information of the corresponding deck layer are extracted from the topology structure of the indoor deck, and a horizontal dimension topology constraint of the candidate deck layer is established; The horizontal coordinates of the target object in the fusion positioning result are projected to the planar connected region corresponding to the candidate deck layer set, the spatial distance deviation of the projection point and the boundary of each connected region is calculated, and according to the spatial distance deviation and the horizontal dimension topology constraint, a target deck layer matched with the horizontal coordinates of the target object is screened out from the candidate deck layer set; The hierarchical identifier of the target deck layer is associated and bound with the three-dimensional space coordinates in the fusion positioning result, and a three-dimensional positioning result containing a deck layer identifier is generated.

[0066] In the embodiment, the vertical coordinate range and the interlayer spacing information of each deck layer are extracted from the cabin structure data, and a vertical dimension constraint model of the deck layer is constructed. Specifically, the cabin structure data is parsed to extract the ground absolute height value and the layer height information of each deck layer, forming a data structure such as {deck layer identifier, bottom height value, top height value}. For example, for a three-deck structure, the deck 1 layer {bottom height 0 meters, top height 2.8 meters}, the deck 2 layer {bottom height 3 meters, top height 5.8 meters}, and the deck 3 layer {bottom height 6 meters, top height 8.8 meters} can be obtained. These data constitute the vertical dimension constraint model, which is used for subsequent deck layer determination.

[0067] After obtaining the three-dimensional space coordinates of the target object in the fusion positioning result, the vertical component of the coordinates is compared with the vertical dimension constraint model. Specifically, the z value in the three-dimensional coordinates (x, y, z) of the target object is extracted, and it is checked whether the z value falls within the height range of each deck layer. For example, if the vertical coordinate z of the target object is 5.2 meters, it can be preliminarily determined that the target is located on the deck 2 layer (within the range of 3 meters to 5.8 meters) by comparing with the vertical dimension constraint model. If the z value of the target is 2.9 meters, the target is located in the transition area between the deck 1 layer and the deck 2 layer, and at this time, both the two adjacent deck layers are included in the candidate deck layer set as candidate levels.

[0068] According to the candidate deck layer set determined in the previous step, the planar connected region and the spatial accessibility information of the corresponding deck layer are extracted from the topological structure of the cabin deck to establish the horizontal dimension topological constraint. For each candidate deck layer, the planar layout is extracted, the boundary contours of the connected regions such as corridors, rooms, and open areas are identified, and the position information of the connection points such as doors and passages is recorded. For example, for the deck 2 layer, the geometric boundaries of all rooms, the center line of the corridor, and the position coordinates of the doors are extracted to construct a connected graph structure representing the spatial connectivity relationship between different regions in the layer.

[0069] The horizontal coordinates of the target object in the fusion positioning result are projected onto the planar connected regions corresponding to the candidate deck layer set. Specifically, the horizontal components (x, y) in the three-dimensional coordinates (x, y, z) of the target object are projected onto the planar layout of each candidate deck layer to obtain the projection point coordinates. The spatial distance deviation between the projection point and the boundary of each connected region is calculated. For example, if the projection point of the target object on the deck 2 layer is located in a certain corridor, it is 0.8 meters away from the corridor boundary; and if the projection point on the deck 1 layer is located inside a wall, it is 1.5 meters away from the nearest passable region, then the target is located on the deck 2 layer.

[0070] According to the calculated spatial distance deviation and horizontal dimension topological constraint, a target deck layer matching the horizontal coordinates of the target object is selected from the candidate deck layer set, a scoring mechanism is set, and a matching score is calculated for each candidate layer, including whether the projection point is located in the passable area, the minimum distance from the boundary of the impassable area, the relative position relationship with the key reference (such as wall, door), etc., and the candidate layer with the highest matching score is selected as the target deck layer. If the scores are close, further judgment can be made in combination with the confidence of the vertical coordinates or the continuity of the historical positioning trajectory.

[0071] The level identifier of the target deck layer is associated and bound with the three-dimensional space coordinates in the fusion positioning result, and a three-dimensional positioning result containing the deck layer identifier is generated. Specifically, the determined deck layer identifier (such as "deck 2 layer") is added to the attribute information of the target object, together with the original three-dimensional coordinates (x, y, z), to form enhanced positioning information {object ID, deck layer identifier, x coordinate, y coordinate, z coordinate, timestamp}, which not only retains the original accurate spatial coordinates, but also provides semantic deck level information, facilitating subsequent cabin navigation, personnel monitoring and other applications.

[0072] In actual application scenarios, for the internal personnel positioning of a large ship, multiple target objects can be tracked simultaneously, for example, ten crew members are detected to be distributed on different deck layers. Through the above method, the specific deck layer where each crew member is located can be accurately identified, and different colors can be marked and displayed in the three-dimensional model of the ship management system. Even in the stair or elevator area between the decks, the transition state of the target can be accurately judged according to the vertical coordinates and horizontal topological constraint, thereby improving the accuracy and reliability of the cabin personnel positioning.

[0073] When the target object moves between the decks, the three-dimensional coordinate changes are continuously tracked, and the deck layer identifier is updated in real time, so that a complete movement trajectory record can be formed. This has important value for cabin emergency evacuation, personnel management and ship safety monitoring, and can quickly locate specific personnel in an emergency and plan the optimal evacuation route.

[0074] The embodiment of the application is a multi-source heterogeneous perception fusion positioning system for a high-dynamic three-dimensional scene in a ro-ro passenger ship cabin, which comprises: A first unit is configured to obtain cabin structure data of the ro-ro passenger ship cabin, six-degree-of-freedom motion parameters of the ship body, and multi-source spatial perception data of a target object. A second unit is configured to decompose the rigid motion component of the ship body according to the six-degree-of-freedom motion parameters of the ship body, and synchronously compensate the multi-source spatial perception data to eliminate the coupling effect of the ship body motion, thereby obtaining the multi-source spatial perception data after decoupling the ship body motion. The third unit is configured to perform feature extraction on the decoupled multi-source spatial perception data, establish a spatio-temporal correspondence relationship between different perception data sources, and obtain an aligned multi-modal feature set; The fourth unit is configured to identify a propagation path anomaly identifier and a signal strength attenuation identifier of each perception data source based on the multi-modal feature set, and calculate a confidence weight of each perception data source. The fifth unit is configured to generate a candidate positioning solution of each perception data source based on the multi-modal feature set, eliminate physically unattainable candidate positioning solutions according to a spatial connectivity constraint of the cabin structure data, combine the confidence weight for weighted fusion, and obtain a fusion positioning result of the target object. The sixth unit is configured to match the fusion positioning result with a topological structure of a cabin deck, distinguish target objects of different deck layers by using a vertical dimension floor constraint, and generate a three-dimensional positioning result containing a deck layer identifier.

[0075] In a third aspect, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0076] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0077] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0078] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for multi-source heterogeneous perception fusion and positioning of high-dynamic three-dimensional scenes inside the cabin of a passenger roll-on / roll-off ship, characterized in that, include: Acquire cabin structure data of passenger roll-on / roll-off ships, six-degree-of-freedom motion parameters of the hull, and multi-source spatial perception data of target objects; Based on the six-degree-of-freedom motion parameters of the hull, the rigid motion components of the hull are decomposed, and the multi-source spatial sensing data is synchronously compensated to eliminate the coupling effect of the hull motion, thus obtaining the multi-source spatial sensing data after the hull motion is decoupled. Feature extraction is performed on the decoupled multi-source spatial sensing data to establish the spatiotemporal correspondence between different sensing data sources and obtain an aligned multimodal feature set. Based on the multimodal feature set, the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source are identified, and the confidence weight of each sensing data source is calculated. Based on the multimodal feature set, candidate localization solutions for each sensing data source are generated. According to the spatial connectivity constraints of the cabin structure data, physically inaccessible candidate localization solutions are eliminated. The solutions are then weighted and fused together with the confidence weights to obtain the fused localization result of the target object. The fused positioning results are matched with the topology of the deck inside the cabin, and the target objects of different deck layers are distinguished by the vertical dimension floor constraints to generate a three-dimensional positioning result containing deck layer identifiers.

2. The method according to claim 1, characterized in that, Based on the six-degree-of-freedom motion parameters of the hull, the rigid motion components of the hull are decomposed, and the multi-source spatial sensing data is synchronously compensated to eliminate the coupling effect of the hull motion, resulting in multi-source spatial sensing data after hull motion decoupling, including: The time series of roll angle, pitch angle and yaw angle in the six degrees of freedom motion parameters of the hull are transformed in the frequency domain. The angle frequency domain components in each angle time series that are within the range of the hull's natural rolling frequency are extracted. The angle frequency domain components are then subjected to inverse Fourier transform to obtain the hull rigid angle components. Based on the frequency characteristics of the rigid angle components of the hull, displacement frequency domain components with the same frequency are identified from the displacement time series of the six degrees of freedom motion parameters. The rigid displacement components of the hull are obtained by inverse Fourier transform. Combined with the rigid angle components of the hull, the rigid motion components of the hull are obtained. Extract the acquisition time of each data frame from the multi-source spatial sensing data, and find the corresponding rigid motion parameters in the rigid motion components of the hull at that time. The rigid motion parameters are converted into three-dimensional rotation transformation operators and three-dimensional translation transformation operators. The inverse transformation of the three-dimensional rotation transformation operator and the inverse transformation of the three-dimensional translation transformation operator are sequentially applied to the spatial sensing data in each data frame to generate multi-source spatial sensing data after decoupling of hull motion.

3. The method according to claim 1, characterized in that, Feature extraction is performed on the decoupled multi-source spatial sensing data to establish spatiotemporal correspondences between different sensing data sources, resulting in an aligned multimodal feature set, including: The original modal features of each sensing data source are extracted from the decoupled multi-source spatial sensing data. The original modal features characterize the spatial distribution features and temporal evolution features of the target object. Obtain the timestamps, spatial coordinate system parameters, sampling frequency, and data format attributes of each sensing data source, and analyze the temporal and spatial heterogeneity of different sensing data sources. To address time domain heterogeneity, the sampling times of each sensing data source are mapped to a unified time base, and a time synchronization relationship across data sources is established through timestamp interpolation; to address spatial domain heterogeneity, the spatial coordinates of each sensing data source are uniformly transformed to the ship's coordinate system, and a spatial alignment relationship across data sources is established. Based on the temporal synchronization relationship and spatial alignment relationship, cross-data source features that are within the same time window and point to the same spatial region are identified, and different modal features that meet the spatiotemporal association conditions are associated and bound to generate an aligned multimodal feature set.

4. The method according to claim 1, characterized in that, Based on the multimodal feature set, the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source are identified, and the confidence weight of each sensing data source is calculated, including: Based on the multimodal feature set, the spatial distribution features and temporal evolution features of each sensing data source representing the target object are extracted. Distance observation values ​​are extracted from the spatial distribution features. The shortest spatial connection path length is determined by combining the connectivity topology of the cabin structure, and the theoretical shortest propagation delay is calculated. The signal arrival time sequence is extracted from the time-series evolution characteristics, and the signal time of each sensing data source is mapped to a unified benchmark. When the actual propagation delay exceeds the propagation delay tolerance of the theoretical shortest propagation delay, an abnormal propagation path identifier is generated. Extract the time series of the received signal strength, analyze the rate of change of signal strength attenuation over time, calculate the theoretical signal strength attenuation value based on the material properties on the shortest spatial connection path, and mark the actual attenuation value as a signal strength attenuation indicator when it exceeds the attenuation tolerance of the theoretical attenuation value. The frequency of occurrence of the propagation path anomaly marker and the signal strength attenuation marker within a preset time window is statistically analyzed, converted into anomaly degree quantification values, and the confidence weight of each sensing data source is calculated.

5. The method according to claim 1, characterized in that, Based on the multimodal feature set, candidate localization solutions from each sensing data source are generated. According to the spatial connectivity constraints of the cabin structure data, physically inaccessible candidate localization solutions are eliminated. These solutions are then weighted and fused using the confidence weights to obtain the fused localization result of the target object, including: Based on the multimodal feature set, the spatial distribution features and temporal evolution features of each sensing data source are extracted, the spatial distribution features of each sensing data source are converted into spatial position estimates in the ship coordinate system, and the search space range is determined. Combined with the motion parameters in the temporal evolution features, candidate localization solutions for each sensing data source are generated in the search space. Based on the cabin structure data, the spatial connectivity topology of the cabin is extracted, a cabin spatial connectivity graph is constructed, the spatial coordinates of each candidate positioning solution are mapped to the nodes of the cabin spatial connectivity graph, the spatial unit node to which each candidate positioning solution belongs is determined, and candidate positioning solutions belonging to physically unreachable isolated nodes are eliminated to obtain valid candidate positioning solutions. Within a continuous time window, the spatiotemporal trajectory of valid candidate localization solutions is extracted, and the spatial displacement at adjacent time points is calculated. When the displacement exceeds the motion range predicted based on temporal evolution features, it is marked as a motion anomaly candidate localization solution. Based on the confidence weight, the valid candidate localization solutions not marked as motion anomalies are weighted and fused to obtain the fused localization result of the target object.

6. The method according to claim 5, characterized in that, The spatial distribution characteristics of each sensing data source are converted into spatial position estimates in the ship's coordinate system, and the search space range is determined. Combining the motion parameters in the temporal evolution characteristics, candidate localization solutions for each sensing data source are generated within the search space, including: The observation values ​​of each sensing data source are extracted from the spatial distribution characteristics. Based on the measurement model of each sensing data source, the observation covariance of the observation values ​​is calculated and mapped to the scale parameter of the search space. The observation values ​​are mapped to the ship coordinate system to generate the spatial position estimate of each sensing data source. Using the geometric center of the spatial position estimate of each sensing data source as the reference point, the boundary coordinates of the search space are determined according to the scale parameter to construct the candidate position search space. Motion parameters are extracted from the temporal evolution features. The motion parameters include velocity vectors and acceleration vectors. Based on the velocity vectors and acceleration vectors, the position offset of each sensing data source at the next moment is predicted. The position offset is superimposed on the spatial position estimate of each sensing data source to obtain the predicted position of each sensing data source. The predicted position is subjected to spatial constraint determination of the candidate position search space. When the predicted position is within the boundary coordinate range of the candidate position search space, the predicted position is used as a candidate positioning solution. When the predicted position exceeds the boundary coordinate range, the predicted position is projected onto the boundary, and the projected position is used as a candidate positioning solution.

7. The method according to claim 1, characterized in that, The fused positioning results are matched with the topology of the decks inside the cabin. Vertical floor constraints are used to distinguish target objects on different deck levels, generating a 3D positioning result containing deck level identifiers, including: Extract the vertical coordinate range and inter-layer spacing information of each deck layer from the cabin structure data, construct a vertical dimension constraint model of the deck layer, obtain the three-dimensional spatial coordinates of the target object in the fusion positioning result, compare the vertical component of the three-dimensional spatial coordinates with the vertical dimension constraint model, and determine the candidate deck layer set to which the target object belongs. Based on the candidate deck layer set, extract the planar connectivity region and spatial accessibility information of the corresponding deck layer from the topology of the cabin deck, and establish the horizontal dimension topological constraints of the candidate deck layer. The horizontal coordinates of the target object in the fusion positioning result are projected onto the planar connected region corresponding to the candidate deck layer set. The spatial distance deviation between the projection point and the boundary of each connected region is calculated. Based on the spatial distance deviation and the horizontal dimension topological constraint, the target deck layer that matches the horizontal coordinates of the target object is selected from the candidate deck layer set. The hierarchical identifier of the target deck layer is associated and bound with the three-dimensional spatial coordinates in the fused positioning result to generate a three-dimensional positioning result containing the deck layer identifier.

8. A high-dynamic three-dimensional scene multi-source heterogeneous perception fusion positioning system for passenger roll-on / roll-off ship cabins, used to implement the method as described in any one of claims 1-7, characterized in that, include: The first unit is used to acquire cabin structure data of passenger roll-on / roll-off ships, six-degree-of-freedom motion parameters of the hull, and multi-source spatial perception data of the target object; The second unit is used to decompose the rigid motion components of the hull based on the six degrees of freedom motion parameters of the hull, and to synchronously compensate the multi-source spatial sensing data to eliminate the coupling effect of the hull motion and obtain the multi-source spatial sensing data after the hull motion is decoupled. The third unit is used to extract features from the decoupled multi-source spatial sensing data, establish the spatiotemporal correspondence between different sensing data sources, and obtain an aligned multimodal feature set. The fourth unit is used to identify the propagation path anomaly identifier and signal strength attenuation identifier of each sensing data source based on the multimodal feature set, and to calculate the confidence weight of each sensing data source. The fifth unit is used to generate candidate localization solutions for each sensing data source based on the multimodal feature set, eliminate physically inaccessible candidate localization solutions according to the spatial connectivity constraints of the cabin structure data, and perform weighted fusion based on the confidence weight to obtain the fused localization result of the target object. The sixth unit is used to match the fused positioning results with the topology of the deck inside the cabin, use the vertical dimension floor constraints to distinguish target objects of different deck layers, and generate a three-dimensional positioning result containing deck layer identifiers.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images