Method and system for dynamically identifying ship living room personnel and updating name plate based on computer vision

Abstract

The invention relates to the technical field of ship Internet of Things, in particular to a living cabin personnel dynamic identification and name plate updating method and system based on multi-modal vision. A wide-angle camera array is deployed in a residence cabin door forbidden area to collect RGB / infrared video streams, a lightweight YOLOv7 model is adopted to detect personnel targets, improved ResNet-18 is combined to extract rotation invariant features, Mahalanobis distance is utilized to calculate matching confidence, a matching threshold value theta < match > is dynamically adjusted, and finally a virtual-real fusion name plate is generated through a perspective transformation matrix. The technical problems of feature drift, identity updating delay and the like caused by insufficient multi-view coverage of a ship environment and motion blur are solved, and 98.3% recognition accuracy and 0.5 second updating delay are realized. The system comprises an explosion-proof camera module, an edge calculation unit and an electronic name plate display module, and meets the real-time personnel management requirement in a harsh ship environment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer system engineering, and particularly relates to a ship cabin personnel dynamic identification and nameplate updating method and system based on computer vision. BACKGROUND

[0002] The existing ship cabin personnel management technology has three defects:

[0003] For example, the RFID tag system adopted by patent CN1121325A requires crew members to wear electronic tags, which infringes on privacy and the tags are easily damaged (the rate of crew members forgetting to take off the tags when leaving the ship is as high as 17%); and the bed pressure sensing scheme (JP2020150001A) cannot detect standing personnel, and the installation requires modification of the cabin structure, increasing the cost by more than 40%, which indicates that the sensor scheme is limited.

[0004] The single camera scheme in the document "Research on Ship Intelligent Monitoring System" (2023) has a misidentification rate of more than 15% under ship sway conditions, as it does not consider image blurring caused by rolling motion, and traditional feature matching methods (such as SIFT) fail in low-light environments, which indicates that the visual scheme has poor environmental adaptability.

[0005] The existing electronic nameplate system (such as DE102022003789A1) relies on manual input, with an average update delay of 120 seconds, which cannot meet the requirements of the IMO "Ship Safety Management System" for real-time cabin personnel status recording, indicating that the system has a display identity update delay problem.

[0006] The root cause of the problem lies in the insufficient multi-angle coverage of the closed environment of the ship, the feature drift caused by dynamic motion, and the lack of intelligent decision-making ability in pure hardware schemes. SUMMARY

[0007] In view of the defects in the prior art, the present application provides a ship cabin personnel dynamic identification and nameplate updating method based on computer vision, comprising the following steps:

[0008] Step S101, deploying a wide-angle camera array in the cabin access control area, synchronously collecting RGB and near-infrared video streams, and transmitting them to an edge computing unit through the ship Ethernet;

[0009] Step S103, using a lightweight YOLOv7 model to detect personnel targets in the video in real time, and the model only outputs the bounding box B i of the detection confidence po≥0.85, where i represents the feature index of the registered personnel in the historical database;

[0010] Step S105, for each B i region, using an improved ResNet-18 network to extract a deep feature vector f i representing a feature vector of a historical personnel i;

[0011] Step S107, calculating the matching confidence of the current frame personnel and the historical database record: where D m is the Mahalanobis distance, φ(·) is a feature projection function, σ is a bandwidth parameter, and j represents the index of the newly monitored personnel feature in the current video frame;

[0012] Step S109, if max(C ij )≥θ match , update the corresponding nameplate information; otherwise, assign a new ID and enter the database, θ match According to the ship swing amplitude dynamic adjustment;

[0013] Step S1011, generating a virtual nameplate containing personnel ID, department and access time, and superimposing it to the video stream through the perspective transformation matrix H.

[0014] The lightweight YOLOv7 model in step S103 is compressed through knowledge distillation, and the teacher model is YOLOv7-w6.

[0015] The improved ResNet-18 network in step S105 includes embedding a coordinate attention mechanism in the last residual block of the network.

[0016] In step S107, the feature projection function φ(·) is dynamically generated using a learnable parameter matrix:

[0017] φ(f)=W p ·f+b p , where W p represents a learnable projection matrix with a dimension of d×512, d is the dimension of the target feature subspace, and b p represents a bias vector with a dimension of d.

[0018] The learnable projection matrix W p is optimized through the intra-class and inter-class distance ratio of the ship personnel library, and the following formula is used for optimization:

[0019] where are the positive and negative sample features respectively, ∈ is a smoothing factor, p represents projection, N is the total number of samples, and k represents the sample index.

[0020] In step S107, when matching, the C ij values of at least two cameras for the same personnel are calculated synchronously, and the weighted average value is taken as the final confidence, and the weight is inversely proportional to the spatial angle between the camera and the personnel.

[0021] Wherein, in the step S109, θ match is negatively correlated with the ship roll angular velocity ω, and is expressed by the following formula: θ match = θ0- γ·tanh(w / w0),

[0022] Wherein γ is the sensitivity coefficient, ω0 is the angular velocity threshold, and the value of ω is obtained in real time by the gyroscope.

[0023] Wherein, the method further comprises triggering an audible and visual alarm and locking the hatch when the same ID in-out time interval Δt ≤ T min .

[0024] Wherein, the method further comprises collecting difficult sample every month, and fine-tuning the model through gradient clipping algorithm, wherein the difficult sample comprises a sample with IoU < 0.3 or C ij <0.5.

[0025] Wherein, the method further comprises performing camera calibration and light equalization verification at 0 o'clock every day, and automatically alarming when the calibration error exceeds δ.

[0026] The application also provides a ship cabin personnel dynamic identification and nameplate updating system based on computer vision, comprising:

[0027] A multi-modal acquisition module comprising an explosion-proof wide-angle camera array, an RGB and near-infrared sensor, and supporting H.265 encoding and PoE power supply;

[0028] An edge computing unit comprising an embedded device with Jetson Orin NX, a lightweight YOLOv7 model and an improved ResNet-18 network;

[0029] A dynamic nameplate module comprising an electronic ink screen on the side of the hatch, which receives update instructions through an RS-485 bus, and the refresh delay is ≤ 0.5 seconds;

[0030] A ship environment adapter integrating a six-axis gyroscope, a light sensor and a temperature control circuit, which outputs environmental parameters to the edge unit in real time;

[0031] A central management platform deployed on a ship server, comprising a personnel database, a permission management module and an alarm log system, and communicating with the edge unit through a time-sensitive network (TSN).

[0032] Compared with the prior art, the application has the following advantages:

[0033] Full-process closed-loop management, from personnel entering the detection area to nameplate updating being completed within 0.5 seconds, which is 240 times faster than traditional manual updating; the response time of abnormal behavior is ≤ 100 milliseconds, meeting the time requirement of the SOLAS convention for ship security systems.

[0034] The ship is highly adaptable to the environment. It maintains 92.7% recognition accuracy even when the roll angle reaches 15 degrees per second, and the temperature adaptability covers -25°C to 70°C, meeting the deployment requirements in global navigation areas.

[0035] The system robustness is improved. The dual-camera redundant design ensures that single-point failure does not affect system operation, and the local cache mechanism guarantees 72-hour continuous operation after network interruption.

[0036] Maintenance convenience is innovated. The automatic calibration program shortens the maintenance time to 1 / 5 of the traditional solution, and the model incremental update does not require shutdown, with online parameter optimization. BRIEF DESCRIPTION OF DRAWINGS

[0037] The above and other objects, features and advantages of the present disclosure exemplary embodiments will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which several embodiments of the present disclosure are illustrated by way of example and not limitation, in which like references refer to like elements, and in which:

[0038] Figure 1 is a flow chart showing a computer vision-based ship cabin personnel dynamic recognition and nameplate updating method according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0040] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Plural" generally includes at least two.

[0041] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present application to describe …, these … should not be limited by these terms. These terms are only used to distinguish … from each other. For example, without departing from the scope of the embodiments of the present application, the first … can also be referred to as the second …, and similarly, the second … can also be referred to as the first ….

[0042] It should be understood that the term "and / or" as used herein merely describes an associated relationship between associated objects, and can represent three relationships, for example, A and / or B, which can represent three cases of A existing alone, A and B existing simultaneously, and B existing alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0043] Depending on the context, the word "if" as used herein can be interpreted as meaning "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted as meaning "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0044] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that a product or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such product or device. Without more limitations, the element defined by the sentence "comprising a" does not exclude the presence of another identical element in the product or device comprising the element.

[0045] The optional embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0046] Embodiment one,

[0047] As Figure 1 shown, the present application discloses a computer vision-based ship cabin personnel dynamic identification and nameplate updating method, comprising the following steps:

[0048] Step S101, deploying a wide-angle camera array in the cabin access control area, synchronously collecting RGB and near-infrared video streams, and transmitting them to an edge computing unit through a ship Ethernet;

[0049] Step S103, using a lightweight YOLOv7 model to detect personnel targets in the video in real time, and the model only outputs the bounding box B i set, where i represents the feature index of the registered personnel in the historical database;

[0050] Step S105, for each B i region, using an improved ResNet-18 network to extract a deep feature vector f i representing the feature vector of the historical personnel i;

[0051] Step S107, calculate the matching confidence of the current frame personnel and the historical database record: Wherein D m is the Mahalanobis distance, phi(·) is a feature projection function, sigma is a bandwidth parameter, and j represents the index of the newly monitored personnel feature in the current video frame;

[0052] Step S109, if max(C ij ) >= theta match , then update the corresponding nameplate information; otherwise, assign a new ID and enter the database, theta match is dynamically adjusted according to the ship swing amplitude;

[0053] Step S1011, generate a virtual nameplate containing personnel ID, department and access time, and superimpose it on the video stream through the perspective transformation matrix H.

[0054] Embodiment two,

[0055] The present application provides a ship cabin personnel dynamic identification and nameplate updating method based on computer vision, comprising the following steps:

[0056] Step S101, deploy a wide-angle camera array in the cabin access control area, synchronously collect RGB and near-infrared video streams, and transmit them to the edge computing unit through the ship Ethernet;

[0057] Step S103, use a lightweight YOLOv7 model to detect personnel targets in the video in real time, and the model only outputs the bounding box B i set where p i >=0.85, wherein i represents the feature index of the registered personnel in the historical database;

[0058] Step S105, for each B i region, use an improved ResNet-18 network to extract a deep feature vector f i representing the feature vector of the historical personnel i;

[0059] Step S107, calculate the matching confidence of the current frame personnel and the historical database record: Wherein D m is the Mahalanobis distance, phi(·) is a feature projection function, sigma is a bandwidth parameter, and j represents the index of the newly monitored personnel feature in the current video frame;

[0060] Step S109, if max(C ij ) >= theta match , then update the corresponding nameplate information; otherwise, assign a new ID and enter the database, theta match is dynamically adjusted according to the ship swing amplitude;

[0061] Step S1011, generate a virtual nameplate containing personnel ID, department, and access time, and superimpose it on the video stream through perspective transformation matrix H.

[0062] In step S103, the lightweight YOLOv7 model is compressed through knowledge distillation, and the teacher model is YOLOv7-w6.

[0063] In an embodiment, a lightweight YOLOv7 model is used to detect personnel targets in real time in a video. The model input size is 640x640, and only the bounding box B i set, where the model is compressed through knowledge distillation, and the teacher model is YOLOv7-w6.

[0064] YOLOv7 is a high-efficiency real-time target detector that balances speed and accuracy well. YOLOv7 improves detection performance by optimizing network structure and introducing a trainable "bag-of-freebies" method. For example, YOLOv7 achieves an AP (average precision) of 56.8% at 30FPS or higher on GPU V100. In addition, YOLOv7's input processing usually adjusts images to a size of 640x640 to ensure the efficiency and accuracy of the model.

[0065] To further improve the real-time performance of the model, the YOLOv7 model is compressed through knowledge distillation technology. Knowledge distillation is a model compression technique that migrates the knowledge of a complex model (teacher model) to a lightweight model (student model), thereby reducing the computational load and parameter quantity of the model while maintaining high accuracy. In this case, the teacher model is YOLOv7-w6, which is an optimized variant of YOLOv7 with higher performance and lower computational requirements.

[0066] In the target detection task, the confidence threshold is used to filter out low-confidence detection results. In this case, the confidence threshold is set to 0.85, which means only detection results with a confidence higher than or equal to 0.85 will be retained. This setting helps improve the accuracy of detection and reduces false positives and false negatives. YOLOv7's output includes the coordinates of the bounding box and the confidence, and by setting the threshold, high-quality detection results can be effectively screened out.

[0067] One of the design goals of YOLOv7 is to achieve efficient real-time detection. In practical applications, YOLOv7 can maintain high accuracy while achieving fast inference speed. For example, YOLOv7 can achieve an inference speed of 160FPS on GPU V100, far exceeding other real-time object detection models. In addition, the lightweight version of YOLOv7 (such as YOLOv7-tiny) can also meet the needs of real-time detection, suitable for resource-constrained scenarios.

[0068] In a certain embodiment, the YOLOv7 model embeds the lightweight bottleneck structure of MobileNetV3 in the backbone network, replacing 3×3 convolution with depthwise separable convolution.

[0069] MobileNetV3 is a lightweight convolutional neural network architecture based on neural architecture search (NAS), which reduces the number of model parameters and computational load by introducing depthwise separable convolution and inverted residual structure. These designs make MobileNetV3 maintain high detection performance while significantly reducing computational load, making it very suitable for resource-constrained environments.

[0070] In YOLOv7, the backbone network is replaced by MobileNetV3, whose bottleneck structure consists of multiple convolution layers, including 1×1 convolution layers, depthwise separable convolution layers, and pooling layers. Each convolution layer is followed by a nonlinear activation function (such as ReLU or h-swish) to introduce nonlinear characteristics.

[0071] Depthwise separable convolution is one of the core technologies of MobileNetV3, which decomposes the traditional 3×3 convolution operation into two steps: depthwise convolution and pointwise convolution. Depthwise convolution performs independent convolution operations on each channel, while pointwise convolution connects features of different channels. This design can significantly reduce the number of model parameters and computational load while maintaining good feature extraction capabilities.

[0072] In YOLOv7, the traditional 3×3 convolution is replaced by depthwise separable convolution, further reducing the complexity and computational requirements of the model. This replacement not only improves the inference speed of the model, but also reduces memory usage, making it more suitable for running on mobile devices and embedded systems.

[0073] By integrating MobileNetV3 into YOLOv7, the model maintains its original advantages while achieving higher efficiency and resource utilization. The lightweight nature of MobileNetV3 allows it to significantly reduce computational load while maintaining high detection accuracy, making it excel in resource-constrained environments.

[0074] The improved ResNet-18 network in step S105 includes embedding a coordinate attention mechanism in the last residual block of the network.

[0075] Embedding a coordinate attention mechanism (CA) in the last residual block is a strategy to enhance model performance. This mechanism combines spatial position information with channel information, allowing the model to more effectively focus on key regions in the image, thereby improving feature extraction and classification / detection / segmentation task performance.

[0076] In residual networks, residual blocks introduce shortcut connections to alleviate gradient vanishing and exploding problems and allow the network to maintain good performance while increasing depth. Embedding a coordinate attention mechanism in the residual block can further enhance model performance. Specifically, the CA mechanism can be embedded in the last operation of the residual block, i.e., inserting a CA module before the output layer of the residual block to enhance feature representation.

[0077] The embedding method specifically includes:

[0078] The input feature map is the feature map output by the previous layer, which is processed by the residual block and then input to the next layer.

[0079] The CA coordinate attention module first performs average pooling on the input feature map to obtain horizontal and vertical feature maps. Then, these two feature maps are concatenated and reduced in dimension through a 1x1 convolution operation, followed by a nonlinear activation function and a BatchNorm layer for normalization, finally generating attention weights in both directions.

[0080] The generated attention weights are weighted with the original feature map to obtain the weighted feature representation. This process can enhance the model's focus on key regions, thereby improving feature extraction and classification / detection / segmentation task performance.

[0081] In step S107, the feature projection function φ(·) dynamically generates using a learnable parameter matrix:

[0082] φ(f)=W p ·f+b p where Wp represents a learnable projection matrix with dimension d x 512, d is the target feature subspace dimension, b p represents a bias vector with dimension d.

[0083] learnable projection matrix W p By optimizing the intra-class and inter-class distance ratio of the ship personnel library, the following formula is used for optimization:

[0084] where are the positive and negative sample features, ∈ is the smoothing factor, p represents the projection, N is the total number of samples, and k represents the sample index.

[0085] wherein the C ij value of at least two cameras is calculated synchronously for the same person during matching in step S107, and the weighted average value is taken as the final confidence, and the weight is inversely proportional to the spatial angle between the camera and the person.

[0086] wherein in step S109, θ match is negatively related to the ship roll angular velocity ω, and is represented by the following formula: θ match = θ0- γ·tanh(w / w0),

[0087] where γ is the sensitivity coefficient, ω0 is the angular velocity threshold, and the ω value is obtained in real time by the gyroscope.

[0088] wherein the step S1011 uses the following formula to superimpose the perspective transformation matrix H to the video stream: where K is the camera intrinsic parameter, and k[R|t] is the nameplate plane pose.

[0089] wherein the method further comprises triggering an audible and visual alarm and locking the hatch when the same ID enters and exits within a time interval Δt ≤ T min .

[0090] wherein T min is set dynamically according to the cabin level, and the crew cabin T min = 15 seconds, and the driver cabin T min = 5 seconds.

[0091] This mechanism is mainly applied in access control systems or security control systems, used to prevent unauthorized access or ensure that personnel complete the entry and exit operation within a specified time. The following is a detailed explanation of the mechanism:

[0092] In access control systems, the time interval typically refers to the time range within which a person is allowed to complete an operation without triggering an alarm after entering or leaving an access-controlled area. For example, in some systems, if a person does not leave within a specified time after entering, the system considers it an abnormal situation.

[0093] When the entry and exit time intervals Δt of the same ID (i.e., the same person) are less than or equal to the set minimum time interval T min , the system determines that the behavior may not conform to the normal operation process, triggering an alarm. For example, in some security systems, if a person does not leave within a specified time after entering, the system considers it may attempt to stay or perform other illegal operations.

[0094] When the triggering condition is met, the system immediately starts an audible and visual alarm to remind relevant personnel to pay attention to the abnormal situation. For example, in some access control systems, if a person does not leave within a specified time after entering, the system will emit a continuous beep and light up a red LED light, indicating that the system has entered an alert state.

[0095] After triggering the alarm, the system usually automatically locks the hatch to prevent unauthorized personnel from entering or leaving. For example, in some aviation security systems, if the crew does not respond to the alarm within 15 seconds, the hatch will automatically unlock for 5 seconds and then lock again. This mechanism ensures that in emergency situations, the system can quickly take measures to prevent unauthorized access.

[0096] If during the alarm period, the relevant personnel respond in time (such as entering the correct password or moving the switch), the system will execute the operation according to the instruction (such as unlocking or locking) and stop the alarm. For example, in some systems, if the crew responds to the alarm within 15 seconds, the system will stop the alarm and resume normal operation.

[0097] The method further comprises collecting difficult case samples every month, and fine-tuning the model through gradient clipping algorithm, wherein the difficult case samples include samples with IoU<0.3 or C ij <0.5.

[0098] In the target detection task, difficult case samples usually refer to those with low overlap with the ground truth (IoU<0.3) or low confidence (C ij <0.5). These samples are usually difficult for the model to correctly classify, so special attention is needed during training. By setting the IoU threshold to 0.3, candidate boxes with low overlap with the ground truth can be filtered out and used as difficult case samples for focused training.

[0099] Gradient clipping is a technique used to prevent gradient explosion, especially in deep neural networks. It works by limiting the size of the gradients to prevent instability during training. Specifically, when the L2 norm of the parameter vector exceeds a certain threshold, the gradients are normalized to ensure that they are not too large. During the fine-tuning of a model, gradient clipping helps maintain the stability of the training process, avoiding the issue of model parameters being updated too quickly or too slowly due to excessively large gradients, which can affect the performance of the model.

[0100] After collecting the difficult example samples, the model needs to be optimized through fine-tuning. Fine-tuning generally refers to further training based on an existing model according to new tasks or data sets. In this process, the model will use the gradient clipping algorithm to adjust the parameters to ensure that the model can learn more effectively when dealing with difficult example samples. For example, in reinforcement learning, gradient clipping is used to constrain the distance between the policy and the old policy, thereby improving the effectiveness of the algorithm.

[0101] The goal of fine-tuning the model is to improve its performance on difficult example samples. This usually involves adjusting the model's weights and parameters to better capture the characteristics of difficult example samples. For example, in the supervised fine-tuning (SFT) phase, the model is trained using human-labeled data to improve its performance on specific tasks. In the reinforcement learning phase, the model optimizes its policy through interaction with the environment to further improve performance.

[0102] In one embodiment, the model is fine-tuned using progressive knowledge distillation:

[0103] where F is the feature map of the teacher model and the student model, and λ linearly increases from 0.3 to 1.0 with the training round.

[0104] The core idea of this method is that the student model does not learn from the final output of the teacher model at once, but gradually learns from multiple intermediate checkpoints of the teacher model, achieving more efficient and stable knowledge transfer.

[0105] During the distillation process, the intermediate layer feature maps (Feature Map) of the teacher model and the student model are used to measure the similarity between the two. These feature maps reflect the model's ability to abstractly represent input data at different levels. By minimizing the difference between these feature maps, the student model can learn the structure and representation of the teacher model.

[0106] During training, a parameter λ is introduced, which linearly increases from 0.3 to 1.0. The role of this parameter is to control the balance between the distillation loss and the student model's own loss. In the early stage of training, λ is small, and the student model mainly focuses on imitating the output of the teacher model; as training progresses, λ gradually increases, and the student model begins to rely more on its own learning ability rather than completely relying on the teacher model.

[0107] Progressive distillation forms an implicit curriculum by allowing the student model to learn from different checkpoints of the teacher model at multiple stages. This mechanism enables the student model to gradually adapt to more complex tasks, thereby improving learning efficiency and final performance. For example, in some experiments, the progressive distillation method can help the student model learn faster from a large teacher model and achieve higher accuracy in the early stage of training.

[0108] Compared with traditional "one-shot distillation", progressive distillation introduces more intermediate steps in the training process, allowing the student model to better understand the structure and behavior of the teacher model. For example, in some tasks, the progressive distillation method can significantly reduce the training time of the student model and improve its performance on complex tasks.

[0109] The method further comprises performing camera calibration and illumination equalization verification at 0 o'clock every day, and automatically alarming when the calibration error exceeds δ.

[0110] This mechanism includes the following:

[0111] Camera calibration is a key step in computer vision tasks that ensures the accuracy of image data by determining the internal parameters (such as focal length, pixel size, and lens distortion) and external parameters (such as the position and pose of the camera in space) of the camera. Common calibration methods include chessboard calibration and self-calibration.

[0112] Illumination equalization verification refers to adjusting the brightness and contrast of images to maintain consistent visual effects under different lighting conditions. This process usually involves techniques such as histogram equalization to highlight image details and reduce the impact of uneven lighting. In security monitoring, illumination equalization helps improve image clarity, thereby enhancing recognition and analysis accuracy.

[0113] Automatic alarm mechanism, when the results of camera calibration or illumination equalization exceed the preset threshold δ, the system will automatically trigger an alarm.

[0114] The execution time and frequency, the camera calibration and illumination equalization check are performed at zero o'clock every day. This time point may be chosen to avoid peak hours and ensure sufficient use of system resources. In addition, regular checks can help detect and correct camera drift or changes in lighting, ensuring long-term stability of the system.

[0115] Camera calibration and illumination equalization checks are usually implemented through automated scripts or specialized software. For example, RANSAC algorithm is used for image feature matching to improve the robustness of calibration; histogram equalization technique is used to enhance image contrast to improve the effect of illumination equalization. In addition, the system can also use NTP or PTP protocols to achieve nanosecond-level synchronization of device time, ensuring the accuracy of the check results.

[0116] In step S1011, personnel sensitive information is hidden from unauthorized users during rendering, and the nameplate field is controlled by the ship role permission matrix.

[0117] During system rendering, personnel sensitive information is hidden from unauthorized users, and the visibility of the nameplate field is controlled by the ship role permission matrix. This ensures that only authorized users can access sensitive data.

[0118] During the front-end rendering phase, conditional statements can be used to control the display and hiding of fields. For example, in the Vue framework, the created() lifecycle hook can be used to determine whether to display certain fields based on the user type (e.g., userType). If the user type is "EX", certain fields (e.g., supplyCol) can be removed from the table columns, thus hiding the fields. This approach is suitable for scenarios where the display content of the interface needs to be dynamically adjusted based on user roles.

[0119] In terms of permission management, role-based access control (RBAC) is often used to define the access rights of different roles. For example, a user may have "administrator", "operator" or "viewer" roles, each corresponding to a different level of permission. Through the role-based access control matrix, it can be determined which users can access which data fields, thus achieving access control of sensitive information. For example, in a ship management system, "captain", "engineer" and "ordinary crew" roles can be defined, and each role can be assigned different permissions to ensure that only authorized personnel can view or edit sensitive information.

[0120] To further protect sensitive information, data masking technology can be used. Data masking refers to replacing sensitive parts of real data with false or partially hidden information to prevent unauthorized users from obtaining real data. For example, sensitive information such as ID numbers and phone numbers can be partially masked, showing only part of the data (e.g., "****1234"), thus protecting the privacy of users. This way can effectively reduce the risk of sensitive information leakage without affecting the business logic.

[0121] The method further comprises reducing the frame rate of step S101 to 8fps and turning off the infrared fill light when the ship vibration sensor detects the berthing state.

[0122] The purpose of this operation is to reduce unnecessary data collection and energy consumption when the ship is in a stationary state, while avoiding the infrared fill light causing disturbance to the environment at night or in low light conditions.

[0123] From a technical point of view, ship vibration sensors are usually used to monitor the structural health of the ship. In the berthing state of the ship, the system detects that the vibration amplitude is below the danger threshold, so it enters a low-power mode, reducing the frame rate to 8fps to reduce the data collection frequency, thus reducing power consumption and prolonging the service life of the device.

[0124] The turning off of the infrared fill light is related to the ambient light conditions. During the berthing period of the ship, if the ambient light is dark, the infrared fill light is usually used to enhance the clarity of images or videos. However, when the system detects that the ship is in a berthing state, it may mean that the current environment is already bright enough, or additional fill light is not needed. In addition, the infrared fill light may cause disturbance to the surrounding environment when used at night, especially in sensitive areas such as ports, fishing ports, etc., so turning off the infrared fill light helps to reduce the impact on the environment.

[0125] Embodiment three,

[0126] An embodiment of the present application is as follows:

[0127] Step 1: Multispectral video acquisition and environmental perception

[0128] A wide-angle explosion-proof camera is installed on the top of the cabin access control area, which synchronously collects RGB visible light video stream and near-infrared video stream. When the light sensor detects that the cabin illumination is lower than 50 lux, it automatically switches to near-infrared imaging mode and starts the fill light device, while adjusting the gamma correction parameter of the image processing pipeline to 2.2 to enhance the dark details. The six-axis gyroscope collects ship roll angular velocity data in real time at a frequency of 100 times per second, and when the detected angular velocity exceeds 5 degrees per second, the motion compensation processing flow is triggered.

[0129] Step 2: Personnel target detection and feature extraction

[0130] After receiving the video stream data, the edge computing unit inputs the lightweight YOLOv7 model for real-time target detection. The model first scales the input image to a resolution of 640x640, extracts feature maps through the backbone network, and outputs the coordinates of the personnel bounding box with a confidence score greater than 0.85. For each detected personnel target region, an improved ResNet-18 network is used to extract a deep feature vector. The network embeds a coordinate attention mechanism after the last residual block to enhance the rotational invariance by calculating the spatial position weight of the feature map. Finally, a 512-dimensional normalized feature vector is output.

[0131] Step 3: Dynamic identity matching decision

[0132] The personnel features extracted from the current frame are matched with the historical database records. First, the intersection over union (IoU) of the current target bounding box and the historical records is calculated, and then the Mahalanobis distance between the feature vectors is calculated. The matching confidence is calculated by the following formula:

[0133] Matching confidence = IoU x exp(-Mahalanobis distance2 / (2 x 0.6 2 )).

[0134] The matching threshold is dynamically adjusted according to the real-time roll angular velocity of the ship. When the roll angular velocity increases, the threshold reference value is reduced by a factor of 0.15 milliseconds per degree per second. If the highest confidence exceeds the current threshold, it is determined to be the same person; otherwise, a new identity is assigned and entered into the database.

[0135] Step 4: Nameplate information generation and virtual-real fusion

[0136] For the matched personnel, the name, position, and department information is retrieved from the ship's central database. The nameplate projection pose is calculated through the pre-calibrated camera parameters, and the display mode is dynamically selected according to the distance between the personnel and the camera: when the distance is less than 3 meters, the personnel's name, position, and entry and exit time are displayed completely on the electronic ink screen; when the distance is greater than 3 meters, only the personnel number and department code are displayed. The display content is transmitted to the electronic ink screen on the cabin door side through the RS-485 bus, and the refresh delay is controlled within 0.5 seconds.

[0137] Step 5: Abnormal behavior linkage disposal When the system detects that the same identity repeatedly enters and exits within a set time interval, a graded response is executed according to the cabin level: for the cockpit area, when the entry and exit interval is less than 5 seconds, the electromagnetic door lock is immediately triggered to lock the cabin door, a three-level alarm signal is sent to the ship's bridge, and the event details are recorded in the navigation log; for the crew cabin area, when the entry and exit interval is less than 15 seconds, the audible and visual alarm is triggered, but the cabin door is not locked.

[0138] Step 6: System self-maintenance execution

[0139] The automatic maintenance process is started at 0 o'clock every morning: first, control all the cameras to shoot standard color card images, automatically calibrate the white balance parameters by analyzing the color block deviation value; then, compare the parallax data of the same calibration object of the left and right cameras, and when the horizontal offset exceeds 5 pixels, re-execute the binocular camera calibration program. On the first day of each month, collect historical sample data that fails to match, and fine-tune the model weight through an optimization algorithm that limits the gradient amplitude.

[0140] Take the second officer's cabin of a 30,000-ton oil tanker as an example:

[0141] Install a double camera at 1.8 meters inside the cabin door, with a lens pitch angle of 30 degrees, covering a rectangular area of 2.5 meters wide and 3 meters high.

[0142] When the ship encounters 6-level sea conditions during navigation, the gyroscope detects that the roll angular velocity reaches 8 degrees per second, and the motion compensation module automatically enhances the filter strength to compensate for the motion blurred image caused by the ship's body shaking.

[0143] After the second officer enters the detection area, the system completes feature matching within 0.3 seconds, with a confidence level of 0.92, and retrieves its "second officer / navigation department" information from the database.

[0144] Because the person is 2.1 meters away from the camera, the electronic ink screen displays the "Wang Hai | Second Officer | 14:25:03" information completely.

[0145] When the person enters again 10 minutes later, the system detects that the time interval is normal and does not trigger an alarm.

[0146] As shown in Table 1, a key fault handling mechanism consistent with the embodiment of the present application is provided.

[0147] Table 1

[0148]

[0149]

[0150] Example Four,

[0151] As shown in Table 2, the present application also proposes a ship cabin personnel dynamic identification and nameplate updating system based on computer vision, which includes:

[0152] A multi-modal acquisition module including an explosion-proof wide-angle camera array, an RGB and near-infrared sensor, and support for H.265 encoding and PoE power supply;

[0153] Edge computing unit, including embedded device with Jetson Orin NX, built-in lightweight YOLOv7 model and improved ResNet-18 network;

[0154] Dynamic nameplate module, including electronic ink screen on the side of the hatch, receiving update instructions through RS-485 bus, refresh delay ≤0.5 seconds;

[0155] Ship environmental adapter, integrating six-axis gyroscope, light sensor and temperature control circuit, real-time outputting environmental parameters to edge unit;

[0156] Central management platform, deployed on ship server, including personnel database, permission management module and alarm log system, communicating with edge unit through time sensitive network (TSN).

[0157] Table II

[0158]

[0159]

[0160] Example V,

[0161] The embodiments of the present disclosure provide a non-volatile computer storage medium, which stores computer executable instructions. The computer executable instructions can execute the method steps of the above embodiments.

[0162] Note that the computer readable medium described above can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the disclosure, the computer readable signal medium can include a computer readable program code propagated on or through a computer readable medium, in baseband or as part of a carrier wave. The computer readable signal medium can take a variety of forms, including but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. The computer readable signal medium can be any computer readable medium that can be used to carry or store computer readable program code for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted or received over any suitable medium, including but not limited to, wire, cable, fiber optic, RF (radio frequency), or any suitable combination of the foregoing.

[0163] The computer readable medium described above can be included in the electronic device described above; alternatively, the computer readable medium can exist as a separate entity in which the electronic device is enclosed.

[0164] Computer program code for carrying out operations of the disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0165] The computer program product of the present disclosure can be a computer program embodied on a non-transitory computer readable medium. When the program runs on a computer, the flowchart and / or block diagram in the flowchart and / or block diagram can be implemented.

[0166] The units described in the embodiments of the present disclosure can be implemented by software or hardware. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0167] The above describes the preferred embodiments of the present disclosure, which aims to make the spirit of the present disclosure clearer and easier to understand, and is not intended to limit the present disclosure. Any modification, replacement, improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the appended claims of the present disclosure.

Claims

1. A method for dynamic identification and name tag updating of ship cabin occupants based on computer vision, characterized in that, Includes the following steps: Step S101: Deploy a wide-angle camera array in the restricted area of ​​the living quarters door, simultaneously collect RGB and near-infrared video streams, and transmit them to the edge computing unit via the ship's Ethernet; Step S103: Use a lightweight YOLOv7 model to detect people in the video in real time. The model only outputs bounding boxes B with a detection confidence pi ≥ 0.

85. i Set, where i represents the feature index of registered personnel in the historical database; Step S105, for each B i For the region, a modified ResNet-18 network is used to extract deep feature vectors. f i The feature vector representing historical person i; Step S107: Calculate the matching confidence score between the person in the current frame and the historical database record: Where D m Here, φ is the Mahalanobis distance, φ(·) is the feature projection function, σ is the bandwidth parameter, and j represents the index of the newly detected personnel feature in the current video frame; Step S109, if max(C ij )≥θ match If the answer is yes, update the corresponding brand information; otherwise, assign a new ID and enter it into the database. match Adjust dynamically according to the ship's rolling amplitude; Step S1011: Generate a virtual nameplate containing personnel ID, department, and entry / exit time, and overlay it onto the video stream using a perspective transformation matrix H.

2. The method as described in claim 1, characterized in that, In step S103, the lightweight YOLOv7 model is compressed through knowledge distillation, and the teacher model is YOLOv7-w6.

3. The method as described in claim 1, characterized in that, The improved ResNet-18 network in step S105 includes embedding a coordinate attention mechanism in the last residual block of the network.

4. The method as described in claim 1, characterized in that, The feature projection function φ(·) in step S107 is dynamically generated using a learnable parameter matrix: φ(f)=W p ·f+b p W p This represents the learnable projection matrix, which has a dimension of d×512, where d is the dimension of the target feature subspace, and b... p This represents the bias vector, with dimension d; Learnable projection matrix W p The optimization of the intra-class and inter-class distance ratio of the ship personnel database is achieved using the following formula: in denoted as positive and negative sample features, ∈ as the smoothing factor, p as the projection, N as the total number of samples, and k as the sample index.

5. The method as described in claim 1, characterized in that, During the matching process in step S107, C values ​​from at least two cameras are simultaneously calculated for the same person. ij The weighted average value is taken as the final confidence level, and the weight is inversely proportional to the spatial angle between the camera and the person.

6. The method as described in claim 5, characterized in that, In step S109, θ match It is negatively correlated with the ship's roll rate ω, and can be expressed by the following formula: θ match =θ0-γ·tanh(ω / ω0), Where γ is the sensitivity coefficient, θ0 represents the preset reference threshold, and ω0 is the angular velocity threshold, with the ω value obtained in real time through the gyroscope.

7. The method as described in claim 1, characterized in that, The method also includes situations where the time interval between entries and exits for the same ID is Δt≤T. min At that time, an audible and visual alarm will be triggered and the hatch will be locked.

8. The method as described in claim 1, characterized in that, The method also includes collecting difficult examples monthly and fine-tuning the model using a gradient pruning algorithm. These difficult examples include those with IoU < 0.3 or C0.

5. ij Samples <0.

5.

9. The method as described in claim 1, characterized in that, The method also includes performing camera calibration and illumination equalization verification at midnight every day, and automatically issuing an alarm when the calibration error exceeds δ.

10. A computer vision-based system for dynamic identification and name tag updating of ship cabin occupants, comprising: The multimodal acquisition module includes an explosion-proof wide-angle camera array, RGB and near-infrared sensors, and supports H.265 encoding and PoE power supply. Edge computing units include embedded devices powered by Jetson Orin NX, with a built-in lightweight YOLOv7 model and an improved ResNet-18 network; The dynamic nameplate module includes an electronic ink screen on the side of the cabin door, which receives update commands via an RS-485 bus, with a refresh delay of ≤0.5 seconds; The marine environment adapter integrates a six-axis gyroscope, a light sensor, and a temperature control circuit, and outputs environmental parameters to the edge unit in real time. The central management platform, deployed on the ship's server, includes a personnel database, access control module, and alarm log system, and communicates with edge units via a time-sensitive network (TSN).

Images