A method and system for crew intercom device behavior recognition

Abstract

The application discloses a kind of seafarer intercom equipment behavior identification method and system, it is related to ship safety monitoring field, it includes real-time acquisition including the image of seafarer operation scene of intercom equipment;The image is preprocessed, and the preprocessed image is obtained;Using pre-trained ResNet-50 convolutional neural network, the features of the intercom equipment are extracted from the preprocessed image, and the feature vector of the intercom equipment is output;Based on the feature vector, using multiple pre-trained classification models, the type of the intercom equipment is identified respectively and multiple independent classification results are output, and all the classification results are weighted and fused to obtain the final classification result;If the proportion of the final classification result based on the image obtained from the continuous frame reaches the preset threshold, the final classification result is determined as the type of the intercom equipment.The application has the effect of high identification accuracy.

Description

Technical Field

[0001] This application relates to the field of ship safety monitoring, and in particular to a method and system for recognizing the behavior of crew members' communication devices. Background Technology

[0002] In ship navigation operations, the standardization of crew members' operations is directly related to navigational safety. In some critical operational scenarios, such as berthing and unberthing, engine room equipment maintenance, and deck work in adverse sea conditions, crew members making phone calls may be distracted and cause safety accidents. At the same time, ships have clear regulations on the areas and times for the use of different communication devices.

[0003] Currently, monitoring of crew communication equipment on ships mainly relies on manual inspections, which suffers from limited monitoring range, poor real-time performance, and susceptibility to missed detections and misjudgments. With the development of computer vision and artificial intelligence technologies, image recognition-based automatic monitoring solutions are gradually being applied across various industries. However, crew communication equipment identification technology for maritime scenarios still has many shortcomings: Firstly, the shipboard operating environment is complex, with drastic changes in lighting (such as strong light, backlight, and low light at night), image blurring due to salt spray corrosion, and equipment obstruction, making traditional image recognition algorithms insufficiently robust. Secondly, telephones and walkie-talkies have similar shapes, and some small walkie-talkies resemble mobile phones, making it difficult for traditional feature extraction methods to accurately distinguish them, resulting in low identification accuracy. Therefore, there is an urgent need for a technical solution that can adapt to the complex maritime environment and accurately identify the type and usage status of crew communication equipment to improve the automation level and reliability of ship safety management. Summary of the Invention

[0004] To address the shortcomings of traditional methods in terms of robustness and accuracy, this application provides a method and system for recognizing the behavior of crew communication devices.

[0005] In the first aspect, this application provides a method for recognizing the behavior of crew communication devices, which adopts the following technical solution: S1, real-time acquisition of images of crew work scenes containing communication devices; S2. Preprocess the image to obtain a preprocessed image; S3. Use a pre-trained ResNet-50 convolutional neural network to extract features of the calling device from the preprocessed image and output the feature vector of the calling device; S4. Based on the feature vector, multiple pre-trained classification models are used to identify the type of the calling device and output multiple independent classification results. All the classification results are weighted and fused to obtain the final classification result. S5. If the proportion of the final classification result obtained based on the continuous frames of the image reaches a preset threshold, then the final classification result is determined to be the type of the calling device.

[0006] By adopting the above technical solution, images of crew operation scenarios are acquired in real time and preprocessed, ensuring the quality and consistency of the input data. A pre-trained ResNet-50 convolutional neural network is used to efficiently extract feature vectors of the communication devices, avoiding subjective errors. Furthermore, a weighted fusion mechanism combining multiple pre-trained classification models outputs the final classification result, significantly improving the accuracy and robustness of communication device type identification. Finally, the device type is determined based on the proportion of classification results in consecutive frames, enhancing the stability and reliability of the identification process and reducing the risk of misjudgment. The entire process automates communication device type identification, reducing labor costs and operational complexity, while improving the efficiency and accuracy of real-time monitoring.

[0007] Optionally, step S5 may further include determining whether a preset rule is violated based on the final classification result.

[0008] By adopting the above technical solutions, images of crew operation scenarios are acquired in real time and preprocessed, ensuring the quality and consistency of the input data. A pre-trained ResNet-50 convolutional neural network is used to efficiently extract feature vectors from communication devices, avoiding subjective errors. A weighted fusion mechanism combining multiple pre-trained classification models outputs the final classification result, significantly improving the accuracy and robustness of type recognition. Determining device type based on the classification result ratio of consecutive frame images enhances the stability and reliability of the recognition process and reduces the risk of misjudgment. Simultaneously, by judging whether preset rules are violated based on the final classification result, real-time automatic monitoring of crew operation compliance is achieved, effectively preventing violations and improving safety and operational standardization.

[0009] Optionally, S2 includes performing hybrid noise reduction and distortion correction on the image to obtain a corrected image; performing adaptive histogram equalization enhancement on the corrected image to obtain an enhanced image; extracting the ROI from the enhanced image, locating the crew member target using the YOLOv8 human detection algorithm, and cropping the crew member's head and upper limb regions; thus obtaining a preprocessed image.

[0010] By employing the aforementioned technical solutions, hybrid noise reduction and distortion correction are performed on the image, effectively removing environmental noise and lens distortion interference, ensuring the clarity and geometric accuracy of the input image. Subsequently, adaptive histogram equalization enhancement is performed, significantly improving image contrast and detail visibility, and enhancing robustness under complex lighting conditions. Furthermore, the YOLOv8 human detection algorithm is used to accurately locate crew members and extract the head and upper limb regions, achieving intelligent extraction of regions of interest, avoiding interference from irrelevant backgrounds, and focusing on key areas for behavioral recognition of the communication device. This series of preprocessing steps optimizes the efficiency of subsequent feature extraction and classification, improving the overall accuracy and stability of the recognition.

[0011] Optionally, the training process of the ResNet-50 convolutional neural network in S3 includes constructing a labeled dataset based on image samples of crew operation scenarios containing communication devices under different lighting, occlusion, and pose conditions; training the network based on the labeled dataset using an SGD optimizer and a cosine annealing learning rate strategy; and analyzing the classification error and optimizing the model parameters through a confusion matrix to obtain a pre-trained ResNet-50 convolutional neural network; wherein, the ResNet-50 convolutional neural network embeds a coordinate attention mechanism in the bottleneck layer.

[0012] By employing the aforementioned technical solutions, a labeled dataset of image samples from crew operation scenarios under different lighting, occlusion, and poses was constructed. This ensured that the training data covered various challenges in the real environment, improving the generalization ability and robustness of the recognition and classification model under complex conditions. Combining the SGD optimizer and cosine annealing learning rate strategy to train the network achieved efficient and stable model convergence, avoiding local optima and accelerating the training process. By analyzing classification errors using the confusion matrix and optimizing model parameters, the recognition performance was finely tuned, significantly reducing the misclassification rate and improving the accuracy of feature extraction. Furthermore, a coordinate attention mechanism was embedded in the bottleneck layer of the ResNet-50 convolutional neural network, enhancing the ability to focus on key parts of the communication device in the feature space and optimizing the quality of the feature vectors. The entire training process not only improved the model's adaptability to various dynamic scenarios but also reduced the cost of manual parameter tuning through automated optimization, enhancing the overall reliability, practicality, and real-time monitoring efficiency of the behavior recognition system.

[0013] Optionally, S3 includes inputting the preprocessed image into a pre-trained ResNet-50 convolutional neural network; extracting low-level features of the preprocessed image through the convolutional layers of the ResNet-50 convolutional neural network, and then reducing the feature dimension through the pooling layers of the ResNet-50 convolutional neural network; strengthening the key features of the calling device through the coordinate attention mechanism of the bottleneck layer of the ResNet-50 convolutional neural network and outputting the feature vector of the calling device.

[0014] By adopting the above technical solution, the low-level structural features of the calling device are first captured by the convolutional layer, and then the feature dimension is compressed by the pooling layer, effectively reducing computational redundancy and retaining key information. Furthermore, a coordinate attention mechanism is embedded in the bottleneck layer to dynamically strengthen the feature weights of key areas of the calling device in the image, significantly improving the discriminative power and relevance of the feature vectors. This structural design not only optimizes the accuracy and efficiency of feature extraction but also enhances the perception of local device details, reducing the risk of misidentification due to environmental interference. Simultaneously, the hierarchical processing mechanism reduces the complexity of the recognition model and improves inference speed, enabling the system to have higher response efficiency and adaptability in real-time monitoring scenarios, laying a reliable data foundation for subsequent classification fusion and behavior determination.

[0015] Optionally, step S4 specifically includes inputting the feature vector into multiple pre-trained classification models for classification; each classification model outputs an independent classification result; assigning weights based on the confidence of each independent classification result; calculating the comprehensive confidence of each calling device category using a weighted fusion strategy; and taking the device category with the highest comprehensive confidence as the final classification result.

[0016] By adopting the above technical solution, feature vectors are input in parallel into multiple pre-trained classification models, fully leveraging the differentiated advantages of each model, effectively avoiding blind spots in single-model recognition, and significantly improving classification coverage and fault tolerance. Weights are dynamically allocated based on the confidence level of each independent classification result, giving higher decision weights to highly reliable results, thus achieving intelligent weighted fusion of classification results. Through a comprehensive confidence level calculation mechanism, the collaborative judgment results of multiple models are quantitatively evaluated, ensuring that the final classification result possesses both high confidence and high consistency. This fusion strategy not only strengthens the robustness of type recognition in complex scenarios but also reduces the probability of misjudgment caused by environmental interference through dynamic weight adjustment, significantly improving the overall recognition accuracy and stability of the system. Simultaneously, the automated fusion process reduces the need for manual review, optimizes real-time monitoring efficiency, and provides highly reliable technical support for the safety supervision of crew operations.

[0017] Optionally, S4 further includes replacing multiple pre-trained classification models with a Bayesian deep ensemble model group, the Bayesian deep ensemble model group containing at least three heterogeneous lightweight convolutional neural networks; optimizing the weight allocation of each heterogeneous lightweight convolutional neural network using a quantum annealing algorithm to generate a dynamic uncertainty confidence distribution; performing knowledge distillation based on the dynamic uncertainty confidence distribution to compress the Bayesian deep ensemble model group into a single lightweight network; and using the distilled single lightweight network to output the final classification result and device state confidence.

[0018] By adopting the above technical solution, multiple classification models are replaced with a Bayesian deep ensemble model group containing heterogeneous lightweight convolutional neural networks. This effectively integrates the advantages of diverse model architectures, significantly improving the comprehensiveness of feature representation and anti-interference ability. Quantum annealing is used to dynamically optimize the weight allocation of each model, generating a confidence distribution based on uncertainty assessment. This achieves adaptive decision-making balance in complex scenarios, significantly reducing the risk of misjudgment caused by environmental noise and sample bias. Furthermore, knowledge distillation technology is used to compress the ensemble model group into a single lightweight network, significantly reducing computational resource consumption and inference latency while retaining the collaborative accuracy of multiple models. This design not only enhances the robustness and real-time performance of call device type recognition but also improves the system's tolerance to low-quality input data through dynamic uncertainty modeling, providing highly reliable and low-latency technical support for crew safety monitoring.

[0019] Optionally, S4 also includes: incorporating time series consistency constraints during knowledge distillation; generating pseudo-labels using feature vectors from consecutive frames; optimizing the loss function of knowledge distillation based on the pseudo-labels; and outputting a single lightweight network after distillation.

[0020] By adopting the above technical solution, time-series consistency constraints are incorporated into the knowledge distillation process, effectively strengthening the correlation of feature vectors in consecutive frames and significantly improving the model's adaptability and recognition stability in dynamic scenarios. The mechanism of generating pseudo-labels from consecutive frames fully explores the potential patterns in time-series data, providing high-confidence supervision signals for the distillation process and reducing label dependence. Furthermore, the knowledge distillation loss function is optimized based on pseudo-labels to achieve efficient alignment of model parameters, ensuring that the single lightweight network after distillation inherits the accuracy and temporal continuity modeling capabilities of multi-model fusion. This design not only significantly improves the temporal consistency of call device type recognition and reduces inter-frame misjudgment fluctuations, but also enhances the model's generalization performance in low-quality data scenarios through a pseudo-label self-supervision mechanism. The final output lightweight network combines high accuracy and low latency, providing real-time and reliable decision support for crew safety monitoring systems.

[0021] Optionally, the ROI extraction further includes: locating the crew member's hand joints and elbow joints using a pre-trained joint detection model; determining the usage posture of the communication device using the confidence level of the crew member's hand joints and the bending angle of the crew member's elbow joints; when the usage posture of the communication device is confirmed, generating an auxiliary ROI region based on the crew member's hand joints; merging the auxiliary ROI region with the original head and upper limb region, and performing the ROI feature extraction.

[0022] By adopting the above technical solution, the pre-trained joint detection model accurately locates the joints of the crew member's hands and elbows, providing a crucial spatial coordinate foundation for the posture analysis of the communication device. Combining the confidence level of the hand joints with the elbow flexion angle to dynamically determine the usage posture of the communication device significantly enhances the scene adaptability and accuracy of behavior recognition. Furthermore, based on the posture confirmation results, an auxiliary ROI region is generated and intelligently fused with the original head and upper limb region, achieving refined feature capture of communication device operation behavior and effectively overcoming the limitations of traditional fixed region extraction. This design not only significantly improves the perception of complex operational postures, such as gripping, adjusting, and interacting, but also optimizes the coverage and targeting of feature extraction through a dynamic ROI expansion mechanism, reducing recognition bias caused by posture variations. Simultaneously, the accurate spatial positioning capability provides highly reliable data support for subsequent behavior compliance analysis, comprehensively enhancing the system's robustness, real-time performance, and practicality.

[0023] Secondly, this application provides a behavior recognition system for crew communication devices, including an image acquisition module, an image preprocessing module, a feature extraction module, a classification and recognition module, and a behavior determination module; The image acquisition module is deployed in the crew operation scene to acquire real-time images of the crew operation scene, including those from communication devices; The image preprocessing module, connected to the image acquisition module, is used to obtain preprocessed images; The feature extraction module, connected to the image preprocessing module, has a built-in pre-trained ResNet-50 convolutional neural network. By embedding a coordinate attention mechanism in the bottleneck layer, it extracts the feature vector of the calling device from the preprocessed image. The classification and recognition module, connected to the feature extraction module, is used to output the final classification result and the confidence level of the call device status; The behavior determination module, connected to the classification and recognition module, is used to analyze the proportion of the final classification result in consecutive frame images, confirm the call device type when the threshold is reached, and determine whether the preset rules are violated based on the final classification result.

[0024] Understandably, the crew communication device behavior recognition system provided in the second aspect above is used to execute the method provided in this application. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects in the corresponding method, and will not be repeated here.

[0025] In summary, this application includes at least one of the following beneficial technical effects: 1. By adopting a real-time image acquisition and preprocessing process combined with ROI region optimization extraction technology, the system can automatically capture key visual information of crew communication devices and eliminate environmental interference, effectively solving the problems of low efficiency of manual recognition and misjudgment caused by background noise in the existing technology, thereby achieving efficient and stable initial data processing; 2. By adopting the ResNet-50 feature extraction with embedded coordinate attention mechanism and the classification mechanism of multi-model weighted fusion, combined with the Bayesian deep ensemble optimization strategy, the system significantly improves the accuracy and robustness of call device type identification, effectively solves the problems of classification bias and single model limitation in complex lighting occlusion scenarios, and thus achieves high-confidence behavior recognition results. 3. By adopting a device type confirmation and preset rule violation detection mechanism based on continuous frame ratio determination, and combining time sequence consistency constraints and attitude-assisted ROI expansion, the system enhances the recognition stability and behavior compliance monitoring capabilities in dynamic scenarios. It effectively solves the problems of missed detection and false alarm caused by real-time monitoring response delay and attitude changes in existing technologies, thereby realizing automated protection and resource optimization for crew operation safety. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of the method for recognizing the behavior of crew communication devices provided in Embodiment 1 of this application; Figure 2 This is another flowchart of the method for recognizing the behavior of crew communication devices provided in Embodiment 1 of this application; Figure 3 This is a structural diagram of the crew communication device behavior recognition system provided in Embodiment 1 of this application; Figure 4 This is a flowchart of step S204 of the method for recognizing the behavior of a crew communication device provided in Embodiment 2 of this application. Detailed Implementation

[0028] This application discloses a method and system for recognizing the behavior of crew communication devices, which are described below in conjunction with the appendix. Figure 1 - Appendix Figure 4 This application will be described in further detail.

[0029] Reference Figure 1A method for recognizing the behavior of crew communication devices includes S1, real-time acquisition of images of crew work scenes containing communication devices.

[0030] The term "crew work scenario" refers to specific locations where crew members operate during ship navigation or work, including but not limited to key locations such as the bridge, deck work areas, and engine room equipment maintenance areas. "Communication equipment" refers to communication tools that crew members may hold or use in the work scenario, primarily including mobile phones and walkie-talkies. "Real-time acquisition" refers to continuously capturing dynamic scene images, including crew members and their handheld communication devices, at a preset constant frame rate using image acquisition modules deployed in the crew work scenario, thereby obtaining visual data on crew members' use of communication devices during operations.

[0031] Specifically, high-definition infrared cameras are employed and deployed in key operational areas of the vessel to ensure coverage of the crew's main activity range. These cameras are capable of stable operation within a temperature range of -20°C to 60°C and possess a waterproof rating of at least IP67 to withstand the corrosive effects of marine environments such as salt spray and wave impact. The infrared cameras support dual-mode shooting, using color mode to record scene details during well-lit days and automatically switching to infrared mode at night or in low-light conditions to ensure image clarity. During acquisition, the cameras continuously capture images of the crew's work scene at a fixed frame rate set by the system based on the current environmental conditions, simultaneously recording the precise timestamp of each frame to ensure the acquired image sequence has complete temporal information, providing a temporal basis for subsequent analysis of the continuous status of communication equipment use. The goal of image acquisition is to clearly capture the crew's posture while holding the communication equipment, providing raw input data for subsequent image preprocessing, feature extraction, and classification.

[0032] S2. Preprocess the image to obtain the preprocessed image.

[0033] Preprocessing refers to the technical process of optimizing image quality and focusing on key areas using specific algorithms, aiming to solve problems such as image noise, optical distortion, uneven illumination, and background interference in the maritime environment. Preprocessed images are standardized image data generated after hybrid noise reduction, distortion correction, adaptive enhancement, and ROI extraction operations, focusing on crew targets and possessing high-quality feature representation.

[0034] Specifically, the system sequentially performs hybrid noise reduction on the acquired raw images. This operation combines Gaussian filtering and median filtering algorithms to simultaneously eliminate Gaussian noise and salt-and-pepper noise, ensuring the basic quality of the original images. Next, distortion correction is performed, correcting geometric distortion caused by lens optical characteristics based on the infrared camera calibration parameters to restore spatial accuracy and obtain a corrected image. Then, adaptive histogram equalization is performed, dynamically adjusting local contrast enhancement strategies to optimize overall image clarity and detail visibility while avoiding overexposure, particularly enhancing the visual distinction between communication devices and crew members' limbs, resulting in an enhanced image. Finally, ROI extraction is performed, using the YOLOv8 human detection algorithm to locate crew members in the enhanced image, precisely cropping their head and upper limb regions as key analysis areas, thoroughly removing irrelevant background information, and generating a preprocessed image with uniform size and concentrated targets. This hierarchical processing significantly reduces the impact of environmental interference on recognition accuracy, providing high-quality input for the feature extraction module.

[0035] S3. Use a pre-trained ResNet-50 convolutional neural network to extract features of the calling device from the preprocessed image and output the feature vector of the calling device.

[0036] The pre-trained ResNet-50 convolutional neural network refers to a convolutional neural network based on the ResNet-50 architecture, trained and optimized on a large-scale labeled dataset, possessing the ability to recognize complex maritime environments. The features of the communication equipment refer to the key visual attributes of the communication devices held by the crew, such as telephones and walkie-talkies, including the shape, keypad, antenna, and other components. The feature vector is a high-dimensional numerical representation output by the neural network, with a dimension of 2048, comprehensively representing the feature information of the communication device.

[0037] Specifically, the preprocessed image is input into an improved ResNet-50 convolutional neural network. This improved ResNet-50 network reduces computational complexity by decreasing the number of shallow convolutional kernels to 32. Low-level features of the preprocessed image, such as edges and textures, are extracted in the initial convolutional layers of the network, and then the feature dimensions are compressed through pooling layers to reduce redundancy. A coordinate attention mechanism is embedded in the bottleneck layer. This coordinate attention mechanism combines channel attention and spatial attention modeling to enhance the feature response of key areas of the communication device, such as enhancing the saliency of the walkie-talkie antenna or telephone handset, thereby accurately capturing subtle differences between different communication devices. Finally, the improved ResNet-50 convolutional neural network outputs a high-dimensional feature vector as input for subsequent classification.

[0038] It should be noted that the pre-training process of the improved ResNet-50 convolutional neural network involves constructing diverse labeled datasets, including image samples with different lighting conditions, occlusion levels, and crew poses. Data augmentation techniques such as random cropping, flipping, and brightness adjustment are used to improve generalization. Training employs a stochastic gradient descent optimizer and a cosine annealing learning rate adjustment strategy to accelerate convergence, and uses confusion matrix analysis to optimize model parameters based on classification error. This ensures that the trained model achieves a recognition accuracy of no less than 95% and a recall of no less than 93% in a maritime environment, providing a reliable foundation for real-time recognition.

[0039] S4. Based on feature vectors, multiple pre-trained classification models are used to identify the type of calling device and output multiple independent classification results. All classification results are weighted and fused to obtain the final classification result.

[0040] The feature vectors refer to the 2048-dimensional high-dimensional feature data extracted through the improved ResNet-50 convolutional neural network, comprehensively representing the visual attributes of the calling device. Multiple pre-trained classification models refer to two heterogeneous classifiers: Support Vector Machine (SVM) and Softmax Regression, each designed for different classification characteristics. Independent classification results refer to the calling device type determination output by each classification model after inference on the feature vectors, such as telephone, walkie-talkie, and their respective confidence scores. Weighted fusion refers to the strategy of assigning weights based on the confidence scores output by each classification model and calculating the overall confidence score through weighted fusion. The final classification result refers to the calling device type determined after fusion and its highest overall confidence score.

[0041] Specifically, the feature vectors are first input in parallel into an SVM classifier and a Softmax regression model. The SVM classifier constructs a classification hyperplane based on the principle of structured risk minimization, excelling at handling high-dimensional, small-sample data and effectively distinguishing between similar-looking devices, such as small walkie-talkies and mobile phones. The Softmax regression model outputs the probability distribution of each calling device category through multi-class logistic regression, improving the interpretability of the results. The two classification models output the calling device type determination and corresponding confidence scores, respectively; the SVM confidence score ranges from 0 to 1, and the Softmax output probability score ranges from 0 to 1. Subsequently, a confidence-weighted fusion strategy is used to assign weights to the SVM classification results. Assign weights to the Softmax regression results , The overall confidence level C for each equipment category is calculated by weighted summation. The formula is as follows: ,in, The confidence score of the SVM classifier for this device category is a scalar value between 0 and 1, representing the degree of confidence of the SVM classifier in judging this device category based on the principle of structured risk minimization. This is the probability output of the Softmax regression model for this device category, a probability value between 0 and 1, representing the posterior probability that the Softmax regression model predicts the feature vector belongs to this device category. The system calculates the overall confidence score for all candidate calling device categories, such as telephones and walkie-talkies; then compares these overall confidence scores, identifies the maximum value, and determines the calling device category corresponding to the maximum value as the final classification result. The system can also set a confidence threshold to enhance the reliability of the results, for example, 0.85; when the overall confidence score is ≥0.85, it is classified as a walkie-talkie. In other words, the classification result is confirmed when the highest overall confidence score is not lower than the threshold; otherwise, it is considered uncertain.

[0042] It should be noted that this fusion strategy fully leverages the high generalization ability of the SVM classifier in small sample classification and the stability of the Softmax regression model in probabilistic modeling. For example, when drastic changes in illumination cause fluctuations in the confidence level of the Softmax regression model, the high robustness of the SVM classifier can provide compensation. Conversely, when the appearance of the communication devices is highly similar, the fine-grained probability distribution of the Softmax regression model can assist in the discrimination. Furthermore, the weighting coefficients... , Cross-validation was used to optimize and ensure a balance between classification accuracy and recall on a multi-sample test set.

[0043] S5. If the proportion of the final classification result obtained based on consecutive frame images reaches a preset threshold, then the final classification result is determined to be the type of calling device.

[0044] Reference Figure 2 In this embodiment, step S5 further includes determining whether a preset rule is violated based on the final classification result.

[0045] Among them, continuous frame images refer to a sequence of images of crew work scenes collected continuously at fixed time intervals, exhibiting temporal correlation. The final classification result refers to the communication device type and its corresponding confidence level determined after weighted fusion in step S4. The proportion refers to the ratio of the number of final classification results determined to be the same communication device type in multiple consecutive frames to the total number of frames. The preset threshold refers to the status confirmation critical value set by the system; in this embodiment, it is set to 80% to avoid misjudgment in a single frame. Preset rules refer to the communication device usage specifications formulated for different work scenarios in ship safety management, such as prohibiting the use of private telephones in the bridge and allowing only explosion-proof walkie-talkies during engine room maintenance.

[0046] Specifically, the system receives the final classification results from 10 consecutive image frames, counts the frequency of the same communication device type, and calculates its percentage. If the percentage is not less than 80%, the communication device currently used by the crew is determined to be of that type. If eight or more consecutive frames identify it as a telephone, it is confirmed as a telephone usage status, ensuring the robustness and continuity of status determination. Subsequently, the confirmed communication device type is compared in real time with a preset rule base. If a violation is detected, such as using a telephone in the bridge area, the warning module is immediately triggered, activating the audible and visual alarms. At the same time, the timestamp of the violation event, area information, and corresponding image frames are stored on the local server. If the communication device usage complies with the rules, a compliance record is generated and stored. This process improves the reliability of status determination through continuous frame verification and achieves automated violation monitoring in conjunction with the rule base, providing real-time decision support for ship safety management.

[0047] Reference Figure 3 This is a crew communication device behavior recognition system in the embodiments of this application.

[0048] A crew communication device behavior recognition system includes an image acquisition module 1, an image preprocessing module 2, a feature extraction module 3, a classification and recognition module 4, and a behavior determination module 5.

[0049] The image acquisition module 1 uses an infrared camera 11, with hardware deployed in key areas of the crew's work environment, such as the bridge deck and engine room. It has a wide operating temperature range and is waterproof, and is used to acquire real-time images of the crew's work environment, including communication devices.

[0050] The image preprocessing module 2 uses an embedded image processor 21, which is connected to the infrared camera 11 interface of the image acquisition module 1. It receives raw image data and performs a hybrid noise reduction, distortion correction, adaptive enhancement ROI extraction algorithm to output a preprocessed image.

[0051] The feature extraction module 3 is equipped with an accelerated computing GPU 31, which is hardware connected to the output port of the image preprocessing module 2. It has a built-in pre-trained ResNet-50 convolutional neural network and embeds a coordinate attention mechanism in the bottleneck layer to extract the 2048-dimensional feature vector of the calling device from the preprocessed image.

[0052] The classification and recognition module 4 is deployed in the ship's main control room or an independent AI processor. It is connected to the data bus of the feature extraction module 3 and uses multi-classification models such as SVM and Softmax regression to infer the feature vectors and output the final classification result and the confidence level of the communication device status.

[0053] The behavior determination module 5 integrates a microcontroller 51 and is connected to the communication interface of the classification and recognition module 4. When the proportion of the final classification result of the continuous frame images reaches a preset threshold, the device type of the call is confirmed and the warning device is triggered after determining whether the preset rules are violated.

[0054] The following is a description of another embodiment of this method. Please refer to... Figure 4 This is another flowchart illustrating the method for recognizing crew communication device behavior in this application embodiment.

[0055] Based on the method for recognizing crew communication device behavior described in Embodiment 1, this Embodiment 2 adds some specific implementation methods.

[0056] S204 further includes S2041, replacing multiple pre-trained classification models with a Bayesian deep ensemble model group, which contains at least three heterogeneous lightweight convolutional neural networks. S2042, optimizing the weight allocation of each heterogeneous lightweight convolutional neural network using a quantum annealing algorithm to generate a dynamic uncertainty confidence distribution. S2043, performing knowledge distillation based on the dynamic uncertainty confidence distribution to compress the Bayesian deep ensemble model group into a single lightweight network. S2044, using the distilled single lightweight network to output the final classification result and device state confidence.

[0057] Among them, Bayesian deep ensemble model groups refer to ensemble model frameworks composed of multiple lightweight convolutional neural networks with different structures, which quantify the uncertainty of model predictions through Bayesian inference. Heterogeneous lightweight convolutional neural networks refer to convolutional neural networks with few parameters and diverse network architectures, such as MobileNetV3, ShuffleNetV2, and EfficientNet-Lite, designed specifically for edge computing devices. Quantum annealing is an algorithm that simulates the quantum tunneling effect to solve combinatorial optimization problems, used to find the optimal solution for neural network weight allocation. Dynamic uncertainty confidence distribution refers to the confidence vector generated by Bayesian ensemble, whose value reflects the degree of uncertainty of the model regarding the classification result. Knowledge distillation is a technique for transferring knowledge from complex models to simpler models. Single lightweight network refers to a simplified version of the neural network obtained through distillation, which retains the performance of the ensemble model while reducing computational complexity.

[0058] Specifically, a Bayesian deep ensemble model group is first constructed, integrating at least three heterogeneous lightweight convolutional neural networks as base classifiers. The weight allocation optimization problem of the neural networks is solved using the quantum annealing algorithm. This algorithm models weight allocation as an energy function minimization problem, utilizing quantum fluctuations to escape local optima and determine the optimal contribution ratio of each heterogeneous network in the ensemble. For example, MobileNetV3 has a weight of 0.4, ShuffleNetV2 has a weight of 0.3, and EfficientNet-Lite has a weight of 0.3. Monte Carlo Dropout sampling is performed based on the optimized weights to generate a dynamic uncertainty confidence distribution. For example, multiple inferences are performed on the same input, and the output confidence variance reflects the model's uncertainty. Knowledge distillation is then performed, using the dynamic uncertainty confidence distribution as the soft objective of the teacher model. The student network (i.e., a single lightweight network) is guided by the KL divergence loss function to simulate the predictive behavior of the ensemble model. During distillation, a temperature coefficient is incorporated to adjust and soften the objective distribution, ensuring efficient knowledge transfer. Finally, the distilled single lightweight network receives the feature vector input and directly outputs the final classification result and the calibrated call device status confidence. This scheme improves integration efficiency through quantum optimization and enhances classification reliability by utilizing uncertainty modeling. After distillation and compression, it can achieve real-time inference of less than 50ms in ship embedded devices, while maintaining a recognition accuracy of no less than 98% of the integrated model.

[0059] In this embodiment, time series consistency constraints are incorporated into the knowledge distillation process; pseudo-labels are generated using feature vectors from consecutive frames; the loss function of knowledge distillation is optimized based on the pseudo-labels; and a single lightweight network after distillation is output.

[0060] The temporal series consistency constraint refers to the regularization condition that mandates the spatiotemporal coherence of feature vectors from consecutive frames in the latent space during the distillation process. The feature vectors of consecutive frames refer to the 2048-dimensional feature data set extracted from the temporal image sequence by the improved ResNet-50 network in step S3. Pseudo-labels refer to temporary supervision signals generated based on the confidence verification of consecutive frames. The loss function of knowledge distillation is an optimization objective function used to measure the difference in output distribution between the teacher model (a group of Bayesian deep ensemble models) and the student model (a single lightweight network).

[0061] Specifically, in the knowledge distillation stage, a temporal consistency constraint is first applied to the feature vectors of five consecutive frames. The temporal dependencies of the feature vectors are modeled using a bidirectional gated recurrent unit network to generate inter-frame consistency coding vectors. Utilizing the continuous frame verification logic from step S5, when the confidence level of the call device type determination in consecutive frames is consistently higher than 0.8, that call device type is used as a high-reliability pseudo-label. Then, the distillation loss function is optimized, extending the standard KL divergence loss to a composite loss function. The composite loss function = KL divergence loss + λ × consistency constraint loss + μ × pseudo-label cross-entropy loss; where λ is the consistency constraint parameter, typically 0.3; and μ is the pseudo-label supervision parameter, typically 0.5. The student network parameters are simultaneously optimized through backpropagation. Finally, a single lightweight network fusing spatiotemporal characteristics is output.

[0062] It should be noted that the pseudo-label mechanism feeds the subsequent behavior judgment logic forward to the knowledge distillation process, forming a closed-loop optimization. The time constraint ensures the smoothness of prediction when the communication device state changes, such as the transition period when a crew member puts down the walkie-talkie and picks up a mobile phone, eliminating the impact of frame-level recognition jumps.

[0063] In this embodiment, ROI extraction further includes: locating the crew member's hand joints and elbow joints using a pre-trained joint detection model; determining the communication device's usage posture using the crew member's hand joint confidence and elbow joint bending angle; generating an auxiliary ROI region based on the crew member's hand joints when the communication device's usage posture is confirmed; and merging the auxiliary ROI region with the original head and upper limb region to extract ROI features.

[0064] The pre-trained joint detection model refers to a deep learning model optimized based on the OpenPose architecture, specifically designed for human pose estimation in ship operation scenarios. Crew hand joints specifically refer to the coordinates of the crew member's wrist joints and their detection confidence. The crew member's elbow joint flexion angle refers to the angle formed by the elbow joint in three-dimensional space. Communication device usage posture refers to the typical hand-held posture of the crew member, such as raising their hand to talk or holding the device drooping. The auxiliary ROI region refers to the rectangular analysis area extended centered on the hand joints. The original head and upper limb region refers to the key areas of the crew member's upper body captured using the YOLOv8 human detection algorithm.

[0065] Specifically, after completing the basic ROI extraction in YOLOv8, the enhanced image is input into the joint detection model. This model extracts multi-scale human features through a feature pyramid network and uses multi-branch convolutional layers to predict the heatmaps and coordinate offsets of hand joints, specifically the wrist and elbow joints, in parallel. When the confidence score of the hand joints is not lower than 0.85 and the elbow flexion angle is less than 120 degrees, it is determined to be a valid communication device usage posture. Subsequently, a square auxiliary ROI region with a side length of 20 cm is generated centered on the hand joint coordinates. Its actual pixel size is calculated based on the camera calibration parameters. The auxiliary ROI region is then merged with the original head and upper limb region using a bitwise OR operation to form the final analysis region.

[0066] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," "third," and similar terms used in this application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The terms "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "comprising" or "including" and similar terms mean that the elements or objects preceding "comprising" or "including" encompass the elements or objects listed following "comprising" or "including" and their equivalents, and do not exclude other elements or objects. "Above," "below," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0067] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for recognizing the behavior of crew members' communication devices, characterized in that: Includes S1, which captures real-time images of crew members working in the field, including those using communication devices; S2. Preprocess the image to obtain a preprocessed image; S3. Use a pre-trained ResNet-50 convolutional neural network to extract features of the calling device from the preprocessed image and output the feature vector of the calling device; S4. Based on the feature vector, multiple pre-trained classification models are used to identify the type of the calling device and output multiple independent classification results. All the classification results are weighted and fused to obtain the final classification result. S5. If the proportion of the final classification result obtained based on the continuous frames of the image reaches a preset threshold, then the final classification result is determined to be the type of the calling device.

2. The method for recognizing crew communication device behavior according to claim 1, characterized in that: Step S5 also includes determining whether a preset rule is violated based on the final classification result.

3. The method for recognizing crew communication device behavior according to claim 1, characterized in that: S2 includes performing hybrid noise reduction and distortion correction on the image to obtain a corrected image; performing adaptive histogram equalization enhancement on the corrected image to obtain an enhanced image; extracting the ROI from the enhanced image, locating the crew member target using the YOLOv8 human detection algorithm, and cropping the crew member's head and upper limb regions; thus obtaining a preprocessed image.

4. The method for recognizing the behavior of a crew member's communication device according to claim 1, characterized in that: The training process of the ResNet-50 convolutional neural network in S3 includes constructing a labeled dataset based on image samples of crew operation scenarios with communication devices under different lighting, occlusion, and poses; training the network based on the labeled dataset using an SGD optimizer and a cosine annealing learning rate strategy; and analyzing the classification error and optimizing the model parameters through a confusion matrix to obtain a pre-trained ResNet-50 convolutional neural network; wherein, the ResNet-50 convolutional neural network embeds a coordinate attention mechanism in the bottleneck layer.

5. The method for recognizing the behavior of a crew member's communication device according to claim 4, characterized in that: S3 includes inputting the preprocessed image into a pre-trained ResNet-50 convolutional neural network; extracting low-level features of the preprocessed image through the convolutional layers of the ResNet-50 convolutional neural network; and reducing the feature dimension through the pooling layers of the ResNet-50 convolutional neural network. The key features of the calling device are enhanced by the coordinate attention mechanism of the bottleneck layer of the ResNet-50 convolutional neural network, and the feature vector of the calling device is output.

6. The method for recognizing the behavior of a crew member's communication device according to claim 1, characterized in that: S4 specifically includes inputting the feature vector into multiple pre-trained classification models for classification; each classification model outputs an independent classification result; and assigning weights based on the confidence level of each independent classification result. A weighted fusion strategy is used to calculate the overall confidence level for each call device category; The device category with the highest overall confidence level is taken as the final classification result.

7. The method for recognizing the behavior of a crew member's communication device according to claim 6, characterized in that: S4 further includes replacing multiple pre-trained classification models with a Bayesian deep ensemble model group, the Bayesian deep ensemble model group containing at least three heterogeneous lightweight convolutional neural networks; optimizing the weight allocation of each heterogeneous lightweight convolutional neural network using a quantum annealing algorithm to generate a dynamic uncertainty confidence distribution; performing knowledge distillation based on the dynamic uncertainty confidence distribution to compress the Bayesian deep ensemble model group into a single lightweight network; and using the distilled single lightweight network to output the final classification result and device state confidence.

8. The method for recognizing the behavior of a crew member's communication device according to claim 7, characterized in that: S4 also includes: incorporating time series consistency constraints into the knowledge distillation process; generating pseudo-labels using feature vectors from consecutive frames; optimizing the loss function of knowledge distillation based on the pseudo-labels; and outputting a single lightweight network after distillation.

9. The method for recognizing the behavior of a crew member's communication device according to claim 3, characterized in that, The ROI extraction further includes: locating the crew member's hand joints and elbow joints using a pre-trained joint detection model; determining the usage posture of the communication device using the confidence level of the crew member's hand joints and the bending angle of the crew member's elbow joints; when the usage posture of the communication device is confirmed, generating an auxiliary ROI region based on the crew member's hand joints; merging the auxiliary ROI region with the original head and upper limb region, and performing the ROI feature extraction.

10. A behavior recognition system for crew communication devices, characterized in that: It includes an image acquisition module, an image preprocessing module, a feature extraction module, a classification and recognition module, and a behavior determination module; The image acquisition module is deployed in the crew operation scene to acquire real-time images of the crew operation scene, including those from communication devices; The image preprocessing module, connected to the image acquisition module, is used to obtain preprocessed images; The feature extraction module, connected to the image preprocessing module, has a built-in pre-trained ResNet-50 convolutional neural network. By embedding a coordinate attention mechanism in the bottleneck layer, it extracts the feature vector of the calling device from the preprocessed image. The classification and recognition module, connected to the feature extraction module, is used to output the final classification result and the confidence level of the call device status; The behavior determination module, connected to the classification and recognition module, is used to analyze the proportion of the final classification result in consecutive frame images, confirm the call device type when the threshold is reached, and determine whether the preset rules are violated based on the final classification result.

Images