Ship driver behavior recognition method and system based on video and millimeter wave radar

Abstract

The application provides a ship driver behavior recognition method and system based on video and millimeter wave radar, the method comprises the following steps: time synchronization of video image data and three-dimensional space orientation and speed data of the driver, and the data set is formed together with the corresponding driver behavior; processing the video image data to generate an image sequence reflecting the driver behavior characteristics, and processing the three-dimensional space orientation and speed data to generate point cloud data of the driver; constructing a multi-channel feature extraction network to enhance the point cloud data; based on the behavior recognition model of the occupation network, extracting and fusing the features of the image sequence and the point cloud data, and predicting the probability distribution of the driver behavior; training and verifying the behavior recognition model by using the constructed data set; using the trained behavior recognition model to monitor the driver behavior in real time, and improving the recognition accuracy and adaptability.

Description

Technical Field

[0001] This invention belongs to the field of maritime traffic safety technology, specifically relating to a method and system for recognizing ship driver behavior based on the fusion of video and millimeter-wave radar. Background Technology

[0002] Intelligent perception and analysis of ship pilot behavior is crucial for ensuring the safe and stable operation of ships in complex environments. Currently, pilot behavior recognition technology is mainly applied in the field of land vehicles, relying primarily on visual sensors for analysis. However, unlike the relatively stable vehicle environment, ship cockpits are characterized by variable lighting conditions and random pilot positioning, posing numerous challenges to the application of visual sensors.

[0003] In conclusion, although existing ship pilot behavior recognition technologies have made some progress, the complex and ever-changing ship cockpit environment still poses a significant challenge to their application. Therefore, developing a high-precision, stable, and adaptable ship pilot behavior recognition method remains a considerable challenge. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for recognizing ship pilot behavior based on video and millimeter-wave radar, which can improve recognition accuracy and adaptability.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows: As one aspect of the present invention, the present invention provides a method for recognizing ship pilot behavior based on video and millimeter-wave radar, comprising the following steps: S1. Synchronize the driver's video image data with three-dimensional spatial orientation and speed data in time, and together with the corresponding driver behavior, form a dataset; the video image data is collected through a visual sensor, and the three-dimensional spatial orientation and speed data is collected through millimeter-wave radar; S2. Process video image data to generate image sequences that reflect the driver's behavioral characteristics, and process three-dimensional spatial orientation and speed data to generate point cloud data of the driver. S3. Construct a multi-channel feature extraction network to enhance the point cloud data and output the enhanced point cloud data. S4. An action recognition model based on occupancy network extracts and fuses features from image sequences and enhanced point cloud data to predict the probability distribution of driver behavior. S5. Use the constructed dataset to train and validate the behavior recognition model; S6. Utilize the trained behavior recognition model to monitor the driver's behavior in real time.

[0006] According to the above scheme, S1 synchronizes the video image data with the three-dimensional spatial orientation and velocity data through the network time protocol.

[0007] According to the above scheme, the image sequence is specifically obtained through keyframe extraction and image normalization processing.

[0008] According to the above scheme, the point cloud data of S2 is obtained in the following ways: S221. Construct a range-Doppler feature matrix using the collected three-dimensional spatial orientation and velocity data; S222. A static clutter filtering algorithm is used to process the range-Doppler characteristic matrix to remove interference echoes caused by stationary objects inside the ship's cockpit. S223. By processing the three-dimensional data on the virtual channel formed by multiple transmit-receive antenna arrays of the millimeter-wave radar using a high-resolution angle measurement algorithm, range-azimuth features are generated. S224. Based on the range-Doppler feature matrix information, velocity information is obtained, and the range, velocity, and azimuth information are used to generate a preliminary point cloud set; S225. The DBSCAN clustering algorithm is used to process the initial point cloud set, filtering out isolated points located in low-density areas and retaining the clusters formed in high-density areas as the driver's point cloud data.

[0009] According to the above scheme, S3 specifically includes: S301. Divide the driver's point cloud data into spatial grids to generate a multi-channel feature matrix. Extend the multi-channel feature matrix from physical channels to high-dimensional channels using virtual channel extension technology. S302. For each point in the multi-channel feature matrix of the high-dimensional channel, the sine and cosine position coding function is used to embed the three-dimensional coordinates and motion information. Through learnable position weights, the spatial relationship between different channels is associated. S303. The weights of each channel are dynamically calculated through the channel attention mechanism to enhance the point cloud feature response of key parts of the driver. Through cross-channel feature fusion, noisy point clouds are suppressed and the signal-to-noise ratio of weak reflection points is enhanced.

[0010] According to the above scheme, S4 specifically includes: S401. The image sequence is processed using the MobileNet-V3 network to generate visual feature vectors; the enhanced point cloud data is processed using the PointNet network to generate radar feature vectors; the visual feature vectors and radar feature vectors are flattened into one-dimensional vectors respectively. S402. Construct a behavior recognition model based on an occupancy network. Input the flattened visual feature vector and radar feature vector, and model the driver's posture and action features by predicting the occupancy probability of points in three-dimensional space. S403. Using the occupancy probability of points in three-dimensional space as prior knowledge, the flattened visual feature vector and radar feature vector are mapped to a unified 512-dimensional latent space through a linear layer to obtain the implicit feature vectors corresponding to the visual sensor and radar sensor respectively. S404. Based on the implicit feature vectors corresponding to the visual sensor and the radar sensor, two multilayer perceptron decoders with identical structures are selected to generate the probability distributions of the driver's behavior by the visual sensor and the radar sensor, respectively.

[0011] According to the above scheme, in step 403, Euclidean distance metric is introduced to calculate the distance between visual feature vectors and radar feature vectors under the same behavior label, and the loss function is minimized to ensure that visual feature vectors and radar feature vectors are consistent in the latent space.

[0012] According to the above scheme, the multilayer perceptron decoder in step 404 includes: an input layer with 512 dimensions, used to input the implicit feature vectors corresponding to the visual sensor and the radar sensor respectively; two hidden layers, each with 256 units; and an output layer, used to output the probability of various behaviors, including lookout, stationary activity, steering, sitting, and walking.

[0013] According to the above plan, S5 specifically includes: The method employs a parameter-sharing cross-iteration approach, alternating training on data from visual and radar sensors. The two multilayer perceptron decoders share all parameters throughout the training cycle, and through multiple iterations, the model learns general and consistent driver behavior characteristics across sensor types.

[0014] As another aspect of the present invention, the present invention also provides a ship pilot behavior recognition system based on video and millimeter-wave radar, comprising: The dataset construction unit is used to synchronize the driver's video image data with the three-dimensional spatial orientation and velocity data in time, and together with the corresponding driver behavior, constitute the dataset; the video image data is acquired through a visual sensor, and the three-dimensional spatial orientation and velocity data is acquired through millimeter-wave radar; The data preprocessing unit is used to process video image data to generate image sequences that reflect the driver's behavioral characteristics, and to process three-dimensional spatial orientation and velocity data to generate point cloud data of the driver. The data augmentation unit is used to construct a multi-channel feature extraction network to enhance point cloud data and output the enhanced point cloud data. The model building unit is used for behavior recognition models based on occupancy networks to extract and fuse features from image sequences and enhanced point cloud data to predict the probability distribution of driver behavior. The model training unit is used to train and validate the behavior recognition model using the constructed dataset; The real-time monitoring unit is used to monitor the driver's behavior in real time using a trained behavior recognition model.

[0015] The beneficial effects of this invention are as follows: 1. The system deeply integrates video data from visual sensors with point cloud data from millimeter-wave radar to compensate for the performance limitations of a single sensor. A multi-channel feature extraction network is constructed to enhance the point cloud data, significantly improving the spatial resolution and density of the radar point cloud data, enabling the system to capture and identify the driver's subtle movement features. A multimodal fusion model based on occupancy network is designed, which can adaptively and intelligently model video keyframes and enhanced radar point cloud data. Through deep feature correlation analysis, consistent recognition of complex driving behaviors is achieved, thereby improving recognition accuracy and adaptability.

[0016] 2. By adopting a cross-iterative training method based on parameter sharing, the generalization ability of the model under different environmental conditions is significantly enhanced, enabling the system to maintain stable recognition performance in various complex environments. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0018] Figure 2 This is a diagram showing the correspondence between video image data and radar data.

[0019] Figure 3 This is a schematic diagram of the radar data processing flow.

[0020] Figure 4 shows the point cloud before processing by the DBSCAN clustering algorithm.

[0021] Figure 5 The point cloud image is processed by the DBSCAN clustering algorithm.

[0022] Figure 6 This is a schematic diagram of the structure of a millimeter-wave radar angular resolution enhancement model.

[0023] Figure 7 This is a schematic diagram illustrating the function of the channel attention mechanism.

[0024] Figure 8 This diagram illustrates the probability distribution for predicting driver behavior using a behavior recognition model based on occupancy networks. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0026] This invention provides a method for recognizing ship pilot behavior based on video and millimeter-wave radar, such as... Figure 1 As shown, it includes the following steps: S1. Synchronize the driver's video image data with the three-dimensional spatial orientation and speed data in time, and together with the corresponding driver behavior, form a dataset; the video image data is collected through a visual sensor, and the three-dimensional spatial orientation and speed data is collected through millimeter-wave radar.

[0027] The data acquisition platform includes two devices: an Azure Kinect visual sensor and a millimeter-wave radar, which respectively collect image and video data of the driver and millimeter-wave point cloud data. In this embodiment, Network Time Protocol (NTP) is used for time synchronization to ensure the spatiotemporal consistency between the visual image data and the millimeter-wave radar point cloud data.

[0028] During data acquisition, the visual sensor is activated, and video data is recorded at a frame rate of 30 frames per second. At a certain moment... The sensor sends a precise time synchronization request message to the host; and the host receives the message at the specified time. The response begins, and after calculation and processing, at time [time missing]. The response message is sent back to the sensor; finally, the sensor sends a response message at time [time]. The response message was successfully received. and These represent the round-trip delay caused by physical transmission distance and network environment, respectively. Based on the timestamp information in the above interaction process, the time deviation between the sensor and the host is calculated.

[0029] After synchronization, it forms a dataset together with the corresponding driver behavior, such as Figure 2 As shown, video images, radar data, and driver behaviors are correlated one-to-one. Driver behaviors include lookout, stationary movement, steering, sitting, and walking.

[0030] S2. Process video image data to generate image sequences reflecting driver behavior characteristics, and process three-dimensional spatial orientation and velocity data to generate driver point cloud data.

[0031] S21. In order to improve data utilization and extract behavioral features, the video data undergoes the following two preprocessing steps: keyframe extraction and image normalization, and then an image sequence is obtained.

[0032] S22. Process the three-dimensional spatial orientation and velocity data to generate point cloud data of the driver, such as... Figure 3 As shown, the point cloud data is obtained in the following ways: S221. Using the collected three-dimensional spatial orientation and velocity data, construct a range-Doppler feature matrix.

[0033] The acquired radar data is subjected to a Fast Fourier Transform (FFT) to extract key information about the target in terms of range and velocity, constructing a range-Doppler feature matrix. In the fast time dimension, a range-fast Fourier Transform (FFT) is applied to these sampling points to extract the response frequencies at different ranges from the radar echo. A two-dimensional FFT is then performed along the slow time dimension to estimate the frequency of phase changes caused by different velocities at the same range, extracting the Doppler features of the target relative to the radar.

[0034] S222. A static clutter filtering algorithm is used to process the range-Doppler feature matrix to remove interference echoes caused by stationary objects inside the ship's cockpit; a single canceller is used to process the radar signal to cancel adjacent chirp echoes in one pass.

[0035] S223. By processing the three-dimensional data on the virtual channel formed by multiple transmit-receive antenna arrays of the millimeter-wave radar using FFT or other high-resolution angle measurement algorithms, range-azimuth features are generated.

[0036] The CFAR target detection algorithm sets a noise threshold. When the unit that needs to be detected If the target is present in the cell, it is considered to exist; otherwise, it is considered noise.

[0037] S224. Based on the range-Doppler feature matrix information, velocity information is obtained. The range, velocity, and azimuth information are then used to generate a preliminary point cloud set.

[0038] Radar data processing yielded range information about the target. Azimuth Pitch angle Information such as coordinate transformation is used to obtain the coordinates of the detection point in three-dimensional space.

[0039] S224. The DBSCAN clustering algorithm is used to process the initial point cloud set, filtering out isolated points located in low-density areas and retaining the clusters formed in high-density areas as the driver's point cloud data, as shown in Figure 4. Figure 5As shown in Figure 4, which is the point cloud before processing by the DBSCAN clustering algorithm, i.e., the initial point cloud set; Figure 5 The image shows the point cloud after processing with the DBSCAN clustering algorithm. It can be observed that the DBSCAN clustering algorithm retains only the clusters formed by high-density regions as the driver's point cloud data.

[0040] S3. Construct a multi-channel feature extraction network to enhance the point cloud data, and output the enhanced point cloud data, such as... Figure 6 As shown, it specifically includes: 301. Multi-channel feature reconstruction of point cloud data: The driver's point cloud data (including 3D coordinates, velocity, reflection intensity, etc.) is divided into spatial grids to generate a multi-channel feature matrix. The multi-channel matrix can include: altitude, horizontal azimuth, radial velocity, reflection intensity, etc.

[0041] Virtual channel expansion technology extends the multi-channel feature matrix from physical channels to higher-dimensional channels. For example, this virtual channel expansion technology simulates the beam of an antenna array. In this embodiment, the multi-channel feature matrix is ​​expanded from 4 channels to 16 channels.

[0042] Multi-channel feature reconstruction can improve the spatial resolution and feature richness of point cloud data, providing a foundation for subsequent angle enhancement.

[0043] 302. Point cloud location encoding and spatial consistency maintenance: For each point in the multi-channel feature matrix of the high-dimensional channel, a sine and cosine position coding function is used to embed three-dimensional coordinates (x, y, z) and motion information (such as velocity vector).

[0044] By using learnable position weights, spatial relationships between different channels can be associated (such as adjacent channels corresponding to adjacent viewpoints).

[0045] By preserving the geometric structure of the point cloud, spatial information loss can be avoided during feature extraction.

[0046] 303, such as Figure 7 As shown, a channel attention mechanism (such as the SE module) is used to dynamically calculate the weights of each channel, enhancing the point cloud feature response of key driver parts (such as the head and hands). Through cross-channel feature fusion, noisy point clouds are suppressed and the signal-to-noise ratio of weakly reflective points (such as the driver's silhouette behind the car window) is enhanced, thereby increasing the feature correlation between the radar range-Doppler complex matrices of different virtual channels. The enhanced point cloud data is output, significantly improving the recognition accuracy of driver posture and micro-movements.

[0047] Inspired by MAE, the RES model randomly masks input channels and reconstructs missing data. It then utilizes self-supervised learning to transfer model parameters from cascaded radar to single-chip millimeter-wave radar, achieving a practical improvement in angular resolution. The RES model consists of three modules: a position encoding function, a channel attention mechanism, and a Unet-based radar cross-channel feature extraction network. It obtains a two-dimensional range-Doppler complex matrix using cascaded radar equipment and overlays the data from the transmit and receive channels to construct radar range-Doppler complex matrices for 16 virtual channels. A position encoding function is introduced to add position information to each virtual channel. The channel attention mechanism is used to learn the data correlation between different virtual channels. A Unet-based radar cross-channel feature extraction method is employed, utilizing a self-supervised learning strategy. Partially masked channel data is used as training labels, and the model learns to predict unobserved channel data. Finally, the input four virtual channels are concatenated with the model's predicted data from the remaining 12 channels, achieving an angular resolution improvement of up to four times.

[0048] The model introduces positional encoding for a total of 16 two-dimensional distance-Doppler complex matrices in the model input and labels. This invention uses sine and cosine positional encoding functions to process these complex matrices and add a unique position vector to each matrix to maintain spatial consistency during training.

[0049] For multi-channel two-dimensional complex matrix data generated by millimeter-wave radar, a channel attention mechanism is introduced to enhance the feature correlation between different virtual channels.

[0050] S4. A behavior recognition model based on an occupancy network extracts and fuses features from image sequences and enhanced point cloud data to predict the probability distribution of driver behavior. For example... Figure 8 As shown, Specifically, it includes: S401. The image sequence is processed using the MobileNet-V3 network to generate visual feature vectors; the enhanced point cloud data is processed using the PointNet network to generate radar feature vectors; the visual feature vectors and radar feature vectors are flattened into one-dimensional vectors respectively to prepare for subsequent fusion.

[0051] S402. Construct a behavior recognition model based on an occupancy network. Input the flattened visual feature vector and radar feature vector, and model the driver's posture and action features by predicting the occupancy probability of points in three-dimensional space.

[0052] The model includes the following core modules: a sensor feature encoder, which is responsible for extracting features from video image sequences and enhanced radar point cloud data; and a latent space aggregation mechanism, which maps the features of the two sensors to a unified latent space to enhance their correlation.

[0053] The advantage of using a network is that it can process sparse radar point cloud data and combine it with visual information, which significantly improves the robustness of complex behavior recognition.

[0054] 403. Using the occupancy probability of points in three-dimensional space as prior knowledge, the flattened visual feature vector and radar feature vector are mapped to a unified 512-dimensional latent space through a linear layer to obtain the implicit feature vectors corresponding to the visual sensor and radar sensor, respectively.

[0055] Meanwhile, Euclidean distance metric is introduced to calculate the distance between visual feature vectors and radar feature vectors under the same behavioral label. The loss function is minimized to ensure that visual feature vectors and radar feature vectors are consistent in the latent space. When the data of one sensor is interfered with, the other sensor can provide supplementary information.

[0056] 404. Based on the implicit feature vectors corresponding to the visual sensor and the radar sensor, two multilayer perceptron (MLP) decoders with consistent structures are selected to generate the probability distributions of the driver's behavior by the visual sensor and the radar sensor, respectively.

[0057] The multilayer perceptron decoder includes: an input layer with 512 dimensions, used to input the implicit feature vectors corresponding to the visual sensor and the radar sensor respectively; two hidden layers, each with 256 units; and an output layer, used to output the probabilities of various behaviors, including lookout, stationary movement, steering, sitting, and walking.

[0058] The loss function is calculated by training the system to minimize the cross-entropy between the predicted values ​​and the behavior labels. For each sensor, the loss function measures the average log-likelihood error between its predicted probability distribution and the true behavior label.

[0059] For the set MLP architecture, its input layer size is set to 512, containing two hidden layers, each containing 256 neurons. Finally, the output layer outputs the probability distribution of the two sensors' predictions for five predefined driving behaviors.

[0060] S5. Train and validate the behavior recognition model using the constructed dataset, specifically including: The method employs a parameter-sharing cross-iteration approach, alternating training on data from visual and radar sensors. The two multilayer perceptron decoders share all parameters throughout the training cycle, and through multiple iterations, the model learns general and consistent driver behavior characteristics across sensor types.

[0061] S6. Utilize the trained behavior recognition model to monitor the driver's behavior in real time.

[0062] As another aspect of the present invention, the present invention also provides a ship pilot behavior recognition system based on video and millimeter-wave radar, comprising: The dataset construction unit is used to synchronize the driver's video image data with 3D spatial orientation and velocity data in time, forming a dataset together with the corresponding driver behavior. The video image data is acquired through a visual sensor, and the 3D spatial orientation and velocity data is acquired through millimeter-wave radar. The specific construction method is the same as that in S1 above, and will not be repeated here.

[0063] The data preprocessing unit is used to process video image data to generate image sequences that reflect the driver's behavioral characteristics, and to process three-dimensional spatial orientation and velocity data to generate point cloud data of the driver. The specific method is the same as that described in S2 above, and will not be repeated here.

[0064] The data augmentation unit is used to construct a multi-channel feature extraction network to enhance the point cloud data and output the enhanced point cloud data. The specific method is the same as that of S3 above, and will not be repeated here.

[0065] The model building unit is used for behavior recognition models based on occupancy networks to extract and fuse features from image sequences and enhanced point cloud data to predict the probability distribution of driver behavior. The specific method is the same as that of S4 above, and will not be repeated here.

[0066] The model training unit is used to train and validate the behavior recognition model using the constructed dataset. The specific method is the same as that described in S5 above, and will not be repeated here.

[0067] The real-time monitoring unit is used to monitor the driver's behavior in real time using a trained behavior recognition model. The specific method is the same as that of S6 above, and will not be repeated here.

[0068] This invention provides a groundbreaking method for recognizing ship pilot behavior. By innovatively integrating visual sensors with millimeter-wave radar, it overcomes the technical bottleneck of traditional single-sensor systems under complex lighting conditions, achieving continuous and accurate recognition in all weather and all time. It pioneeringly proposes a self-supervised learning-based millimeter-wave radar angular resolution enhancement method, significantly improving the spatial resolution and density of radar point cloud data, enabling the system to capture and recognize the subtle movement features of the pilot. An innovative multimodal fusion model based on occupancy networks is designed, which adaptively and intelligently models video keyframes and enhanced radar point cloud data, achieving consistent recognition of complex driving behaviors through deep feature correlation analysis. A novel cross-iterative training method based on parameter sharing significantly enhances the model's generalization ability under different environmental conditions, ensuring stable recognition performance in various complex environments. This invention provides comprehensive technical protection for safe ship navigation, effectively preventing maritime accidents caused by abnormal pilot behavior. Compared with existing technologies, it represents a qualitative leap and provides a new technical path for the development of intelligent shipping.

[0069] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0070] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for recognizing ship pilot behavior based on video and millimeter-wave radar, characterized in that: Includes the following steps: S1. Synchronize the driver's video image data with three-dimensional spatial orientation and speed data in time, and together with the corresponding driver behavior, form a dataset; the video image data is collected through a visual sensor, and the three-dimensional spatial orientation and speed data is collected through millimeter-wave radar; S2. Process video image data to generate image sequences that reflect the driver's behavioral characteristics, and process three-dimensional spatial orientation and speed data to generate point cloud data of the driver. S3. Construct a multi-channel feature extraction network to enhance the point cloud data and output the enhanced point cloud data. S4. An action recognition model based on occupancy network extracts and fuses features from image sequences and enhanced point cloud data to predict the probability distribution of driver behavior. S5. Use the constructed dataset to train and validate the behavior recognition model; S6. Utilize the trained behavior recognition model to monitor the driver's behavior in real time; The point cloud data for S2 is obtained in the following ways: S221. Construct a range-Doppler feature matrix using the collected three-dimensional spatial orientation and velocity data; S222. A static clutter filtering algorithm is used to process the range-Doppler characteristic matrix to remove interference echoes caused by stationary objects inside the ship's cockpit. S223. By processing the three-dimensional data on the virtual channel formed by multiple transmit-receive antenna arrays of the millimeter-wave radar using a high-resolution angle measurement algorithm, range-azimuth features are generated. S224. Based on the range-Doppler feature matrix information, velocity information is obtained, and the range, velocity, and azimuth information are used to generate a preliminary point cloud set; S225. The DBSCAN clustering algorithm is used to process the initial point cloud set, filtering out isolated points located in low-density areas and retaining the clusters formed in high-density areas as the driver's point cloud data.

2. The method for recognizing ship pilot behavior based on video and millimeter-wave radar according to claim 1, characterized in that: The S1 mentioned above synchronizes video image data with three-dimensional spatial orientation and velocity data through the network time protocol.

3. The method for recognizing ship pilot behavior based on video and millimeter-wave radar according to claim 1, characterized in that: The image sequence is obtained through keyframe extraction and image normalization.

4. The method for recognizing ship pilot behavior based on video and millimeter-wave radar according to claim 1, characterized in that: S3 specifically includes: S301. Divide the driver's point cloud data into spatial grids to generate a multi-channel feature matrix. Extend the multi-channel feature matrix from physical channels to high-dimensional channels using virtual channel extension technology. S302. For each point in the multi-channel feature matrix of the high-dimensional channel, the sine and cosine position coding function is used to embed the three-dimensional coordinates and motion information. Through learnable position weights, the spatial relationship between different channels is associated. S303. The weights of each channel are dynamically calculated through the channel attention mechanism to enhance the point cloud feature response of key parts of the driver. Through cross-channel feature fusion, noisy point clouds are suppressed and the signal-to-noise ratio of weak reflection points is enhanced.

5. The method for recognizing ship pilot behavior based on video and millimeter-wave radar according to claim 1, characterized in that: S4 specifically includes: S401. The image sequence is processed using the MobileNet-V3 network to generate visual feature vectors; the enhanced point cloud data is processed using the PointNet network to generate radar feature vectors; the visual feature vectors and radar feature vectors are flattened into one-dimensional vectors respectively. S402. Construct a behavior recognition model based on an occupancy network. Input the flattened visual feature vector and radar feature vector, and model the driver's posture and action features by predicting the occupancy probability of points in three-dimensional space. S403. Using the occupancy probability of points in three-dimensional space as prior knowledge, the flattened visual feature vector and radar feature vector are mapped to a unified 512-dimensional latent space through a linear layer to obtain the implicit feature vectors corresponding to the visual sensor and radar sensor respectively. S404. Based on the implicit feature vectors corresponding to the visual sensor and the radar sensor, two multilayer perceptron decoders with identical structures are selected to generate the probability distributions of the driver's behavior by the visual sensor and the radar sensor, respectively.

6. The method for recognizing ship pilot behavior based on video and millimeter-wave radar according to claim 5, characterized in that: In step 403, Euclidean distance metric is introduced to calculate the distance between visual feature vectors and radar feature vectors under the same behavioral label. Minimizing the loss function ensures that visual feature vectors and radar feature vectors are consistent in the latent space.

7. The method for recognizing ship pilot behavior based on video and millimeter-wave radar according to claim 5, characterized in that: The multilayer perceptron decoder in step 404 includes: an input layer with 512 dimensions, used to input the implicit feature vectors corresponding to the visual sensor and the radar sensor respectively; two hidden layers, each with 256 units; and an output layer, used to output the probabilities of various behaviors, including lookout, stationary activity, steering, sitting, and walking.

8. The method for recognizing ship pilot behavior based on video and millimeter-wave radar according to claim 7, characterized in that: S5 specifically includes: The method employs a parameter-sharing cross-iteration approach, alternating training on data from visual and radar sensors. The two multilayer perceptron decoders share all parameters throughout the training cycle, and through multiple iterations, the model learns general and consistent driver behavior characteristics across sensor types.

9. A ship pilot behavior recognition system based on video and millimeter-wave radar, characterized in that: include: The dataset construction unit is used to synchronize the driver's video image data with the three-dimensional spatial orientation and velocity data in time, and together with the corresponding driver behavior, constitute the dataset; the video image data is acquired through a visual sensor, and the three-dimensional spatial orientation and velocity data is acquired through millimeter-wave radar; The data preprocessing unit is used to process video image data to generate image sequences reflecting driver behavior characteristics, and to process three-dimensional spatial orientation and velocity data to generate driver point cloud data. The point cloud data is obtained in the following ways: using the collected three-dimensional spatial orientation and velocity data, a range-Doppler feature matrix is ​​constructed; a static clutter filtering algorithm is used to process the range-Doppler feature matrix to remove interference echoes caused by stationary objects inside the ship's cockpit; and a range-azimuth feature is generated by processing the three-dimensional data on the virtual channel formed by multiple transmit and receive antenna arrays of the millimeter-wave radar using a high-resolution angle measurement algorithm. Velocity information is obtained by converting the distance-Doppler feature matrix information. The distance, velocity, and azimuth information are used to generate a preliminary point cloud set. The DBSCAN clustering algorithm is used to process the preliminary point cloud set, filtering out isolated points in low-density areas and retaining the clusters formed in high-density areas as the driver's point cloud data. The data augmentation unit is used to construct a multi-channel feature extraction network to enhance point cloud data and output the enhanced point cloud data. The model building unit is used for behavior recognition models based on occupancy networks to extract and fuse features from image sequences and enhanced point cloud data to predict the probability distribution of driver behavior. The model training unit is used to train and validate the behavior recognition model using the constructed dataset; The real-time monitoring unit is used to monitor the driver's behavior in real time using a trained behavior recognition model.

Images