Intelligent carrier self-service management and control method for multi-modal recognition

Abstract

The application discloses a kind of multi-modal identification's intelligent carrier self-service management and control method, for facing secret management scene to the self-service management and control of intelligent carrier, including the appearance image data of collection intelligent carrier and extract appearance feature, read the RFID identification code of intelligent carrier to obtain carrier metadata, collection intelligent carrier's positioning data, construct and the digital twin model associated with RFID identification code;Before executing warehouse or delivery or handover operation, the face biometric of collection operator and the RFID badge identification code of reading operator are carried out identity check;In the secret management link of carrier control, appearance image data, RFID identification code and positioning data are collected to carry out multi-modal fusion identification to determine carrier identity and carrier state and output allow to pass instruction or locking instruction or alarm instruction, and relevant results are written into digital twin model.The application solves the problem that carrier identity, personnel identity and carrier state are difficult to be consistent in the secret management scene.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a self-management method for intelligent carriers with multimodal recognition. Background Technology

[0002] In government agencies, military research institutes, and other classified locations, classified media are typically used to carry classified information or data. These media include traditional formats such as paper documents and CDs, as well as physical objects such as portable storage media, classified terminals, and detachable storage components. The full lifecycle management of classified media typically involves multiple stages, including registration, storage, borrowing, carrying, returning, transfer, inventory, and destruction. It often involves multi-positional collaboration and cross-regional transfers. Errors in identity verification, inconsistencies in status records, or missing process records can create blind spots in supervision and lead to the risk of leaks. Existing technical documents also point out that problems such as weak management of classified information media, insufficient technical protection capabilities, and a lack of corresponding technical means objectively exist in some units, urgently requiring the improvement of control levels through information technology and equipment. Taking the CN104318360A "Comprehensive Management System for Classified Carriers" as an example, it achieves refined management and daily operation support for classified carriers (such as inventory, registration, transmission, destruction, and borrowing) through a classified carrier database, management and control software, label and QR code system, and external hardware such as barcode scanners, label printers, and card readers.

[0003] On the other hand, the trend towards equipment-based solutions is also advancing self-service and automated storage and retrieval management. Taking CN120106783A, "Security Management Equipment for Classified Carriers," as an example, this solution uses RFID to uniquely identify classified carriers and places them in intelligent control cabinets. Combined with RFID access doors, handheld PDAs, and a back-end management subsystem, it achieves automated recording of entry, exit, and borrowing / return processes, while also supporting self-service storage and retrieval operations through biometric authentication.

[0004] Although existing solutions can improve the efficiency of managing classified materials in terms of identification, ledgers, and access process records, there are still technical challenges that are closely related to security and compliance in real classified scenarios.

[0005] First, the working environment of classified locations may have characteristics such as obstruction, changes in lighting, interference from metal cabinets and shelves, and frequent personnel flow. When a single identification method fails in a part, it is easy to cause problems such as incorrect identification of the carrier, inconsistency between location and status records, and incomplete traceability of the operation chain. This further makes it difficult to detect in a timely manner if classified carriers are misplaced, misclaimed, or carried out of bounds.

[0006] Secondly, there are more subtle adversarial risks in the management of classified carriers. For example, if the static characteristics such as appearance and weight have not changed significantly, the items inside the carrier may be swapped out with equal weight or the internal structure may be replaced. If only RFID tags, simple appearance verification or static weighing results are relied upon, it is often difficult to identify in time that the actual internal contents have changed.

[0007] Furthermore, when classified carriers are overloaded, their storage constraints are altered, or their internal stacking methods are improper, overload and abnormal internal stress may occur, posing safety risks such as the collapse or explosion of closed components. Traditional contact-based inspection methods are often limited by factors such as inspection efficiency, potential damage to the carrier surface, and workstation adaptability, making it difficult to achieve stable coverage and consistent assessments without disrupting the operational rhythm of classified locations. Therefore, how to reliably determine the carrier's identity, status, and key risks throughout the entire process of classified carrier supervision, while ensuring the traceability and auditability of data throughout the entire process, remains a problem that existing technologies need to further address. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a self-management method for intelligent carriers with multimodal recognition, so as to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: A multimodal recognition-based self-service management method for intelligent carriers, used for self-service management of intelligent carriers in classified management scenarios, includes: The appearance image data of the smart carrier is collected and appearance features are extracted. The RFID tag code of the smart carrier is read to obtain carrier metadata. The positioning data of the smart carrier is collected. The appearance features, the carrier metadata and the positioning data are combined to form a digital fingerprint of the carrier, and a digital twin model associated with the RFID tag code is constructed accordingly. Before performing warehousing, warehousing, or handover operations on the smart carrier, the operator's facial biometric features are collected and the operator's RFID badge code is read to perform identity verification. After the identity verification is passed, control operations on the smart carrier are allowed to be initiated. In the process of managing classified carriers during warehousing, warehousing, or handover, the appearance image data, RFID identification code, and positioning data of the smart carrier are collected. Multimodal fusion identification is performed to determine the carrier identity and carrier status of the smart carrier, and an access permission command, a lock command, or an alarm command is output based on the carrier status. The identity verification result, the carrier identity, the carrier status, and the permission instruction, the lock instruction, or the alarm instruction are written into the digital twin model.

[0010] Preferably, the carrier digital fingerprint also includes a dynamic inertia fingerprint for preventing swapping of equal-weight carriers, and the classified carrier control process further includes: controlling an automated transmission mechanism to grab or carry the intelligent carrier, and executing non-uniform speed test actions according to preset motion control commands, the test actions including acceleration, deceleration or rotational maneuvers; during the execution of the test actions, collecting real-time current load data and position feedback data of the drive motor of the automated transmission mechanism, and constructing the dynamic inertia fingerprint based on the real-time current load data and the position feedback data.

[0011] Preferably, the method further includes: retrieving a standard inertia model associated with the RFID tag code or the original dynamic inertia fingerprint from the time of historical warehousing; comparing the dynamic inertia fingerprint with the standard inertia model or the original dynamic inertia fingerprint to obtain a deviation value, wherein the deviation value is the Euclidean distance between the inertia feature vectors of the two; allowing the smart carrier to perform the warehousing, warehousing, or handover operation when the deviation value is not greater than a preset inertia threshold, and outputting the alarm command or the locking command when the deviation value is greater than the preset inertia threshold.

[0012] Preferably, the classified carrier control process further includes a non-contact modal detection step for detecting stress anomalies caused by overload inside the intelligent carrier. The non-contact modal detection step includes: when the intelligent carrier is transferred to the detection station, confirming that the intelligent carrier is in a stationary state and that the physical locking mechanism of its closed component is in the closed position; controlling a pneumatic excitation device to emit a calibration fluid pulse with a preset pressure and preset duration as a transient excitation source to the surface of the closed component; and using a non-contact laser displacement sensor or a Doppler vibration meter to synchronously acquire the micro-displacement time-domain vibration signal of the closed component after being impacted by the calibration fluid pulse.

[0013] Preferably, the method further includes: acquiring a zero-stress reference vibration signal and a dangerous stress reference vibration signal of the same type of intelligent carrier, wherein the dangerous stress reference vibration signal is obtained by performing the calibration fluid pulse excitation on the same type of intelligent carrier under a preset overload level and acquiring it by the non-contact laser displacement sensor or the Doppler vibration meter; comparing the consistency of the micro-displacement time-domain vibration signal with the zero-stress reference vibration signal to obtain the zero-stress consistency degree, and comparing the consistency of the micro-displacement time-domain vibration signal with the dangerous stress reference vibration signal to obtain the dangerous consistency degree.

[0014] Preferably, when the hazard consistency is not less than a preset hazard threshold, the intelligent carrier is determined to be in a high internal pressure overload state and a rejection command and a lock command are output; when the hazard consistency is less than the preset hazard threshold and the zero stress consistency is not less than a preset benchmark threshold, the intelligent carrier is determined to be in a passable state and the passable command is output; when the hazard consistency is less than the preset hazard threshold and the zero stress consistency is less than the preset benchmark threshold, an alarm command or a lock command is output.

[0015] Preferably, the carrier digital fingerprint further includes a feature vector of the appearance feature, which is obtained by performing feature extraction on the appearance image data, and the digital twin model stores the feature vector and the timestamp of the positioning data.

[0016] Preferably, the identity verification includes: comparing the facial biometric features with a pre-registered personnel facial template to obtain a facial comparison result, and matching the RFID work badge identifier code with a pre-registered personnel work badge permission table to obtain a work badge matching result. When the facial comparison result is passed and the work badge matching result is passed, the identity verification is determined to be successful.

[0017] Preferably, the multimodal fusion identification includes: calculating the confidence level of appearance recognition, the confidence level of RFID recognition, and the confidence level of positioning consistency, respectively, and weighting and summing the confidence levels of appearance recognition, RFID recognition, and positioning consistency according to a preset weighting coefficient to obtain a fusion confidence level; outputting the carrier identity and the carrier status when the fusion confidence level is not less than a preset fusion threshold, and outputting the locking command or the alarm command when the fusion confidence level is less than the preset fusion threshold.

[0018] The advantages of this invention over the prior art are: 1) This invention unifies and converges scattered and easily distorted information about the carrier's identity, location, status, and operation into a continuously updated digital fingerprint and digital twin model. In the management of classified carriers, multimodal fusion identification and identity verification serve as prerequisites for entry and release, ensuring consistency verification and recording of every entry, exit, or handover within the same data framework. Addressing the issue of identity verification errors, inconsistent location status, and untraceable operations due to the failure of a single identification method in certain conditions, this invention does not rely on the unwavering reliability of any single sensing method. Instead, it simultaneously collects appearance image data, RFID tag codes, and location data, performing multimodal fusion identification on all three. It outputs permission, locking, or alarm commands based on fusion confidence levels. When a particular modality becomes unstable due to factors such as obstruction, reflection, or metallic interference, other modalities still provide independent constraints, ensuring that the determination of carrier identity and status is no longer based on single-point information. Meanwhile, before initiating control operations, this invention introduces identity verification using facial biometrics and RFID badge codes, and writes the identity verification results, carrier identity, carrier status, and final output instructions into a digital twin model. This ensures that changes in carrier status correspond stably to specific operators, times, and locations, thereby creating a traceable operational chain in the process and avoiding management breakpoints where passage is granted but it is unclear who granted the passage, when it was granted, or why it was granted.

[0019] 2) To address the more subtle risk of weight-based substitution, this invention introduces kinetic inertia fingerprinting as a component of the carrier's digital fingerprint. Its effectiveness stems from a key fact: weight-based substitution can only replicate the scalar quantity of total mass, but it's difficult to simultaneously replicate the spatial distribution of mass and the resulting mechanical properties such as rotational inertia and equivalent inertia. Even if the substitute makes the weight of the replaced item exactly the same, changes in internal structural shape, density distribution, center of gravity position, and the proportion of mass away from the rotation axis will result in different dynamic responses exhibited by the carrier during acceleration, deceleration, or rotational maneuvers. This difference is not dependent on visual appearance or static weighing, but rather on the drive load mode required by the automated transmission mechanism to complete the same non-uniform motion: when the equivalent inertia is larger, a higher drive output is needed to achieve the same trajectory for the same acceleration change; when the equivalent inertia is smaller or the mass distribution is closer to the rotation axis, the drive output mode exhibits a identifiable change. Therefore, this invention controls an automated transmission mechanism to perform non-uniform speed test actions including acceleration, deceleration, or rotational maneuvers, and during the actions, it collects real-time current load data and position feedback data of the drive motor at high frequency. These data are used to form a joint characterization of the drive output and motion response, thereby obtaining a dynamic inertia fingerprint. Since the dynamic inertia fingerprint reflects the impedance characteristics of the carrier and its internal contents to motion, for a swapper to maintain the same weight while ensuring the fingerprint is consistent, they must simultaneously replicate the geometry and mass distribution of the internal items. This is far more difficult in practice than simply making the weight consistent. Therefore, this invention can bring the risk of swapping items of equal weight but different structures into a determinable range, and trigger locking or alarms through deviation values ​​and thresholds, thus preventing abnormalities from entering subsequent warehousing, outbound, or handover operations.

[0020] 3) To address the safety risks of internal overload leading to abnormal stress and potential collapse or explosion of the closed assembly, this invention introduces a non-contact modal detection step. The underlying mechanism is that the structural stiffness, boundary constraints, and energy dissipation characteristics of the closed assembly change under different stress states, resulting in different micro-displacement time-domain vibration responses when subjected to the same transient excitation. Increased stress or overload often indicates that the closed assembly is under stronger tension, compression, or internal pressure constraints, which alters its vibration characteristics after excitation. For example, the initial peak value, vibration energy decay rate, waveform envelope, and similarity of the time-domain response will all show measurable differences from the zero-stress state. Based on this, after confirming that the carrier is stationary and the closed assembly is physically locked in place at the detection station, this invention uses a pneumatic excitation device to emit a calibration fluid pulse with a preset pressure and duration onto the surface of the closed assembly, ensuring the repeatability and comparability of the excitation source across different batches and stations. Then, a laser displacement sensor or Doppler vibration meter is used to collect the micro-displacement time-domain vibration signal, avoiding the installation complexity, surface damage, and cycle time effects associated with contact sensing. Subsequently, the present invention compares the acquired micro-displacement time-domain vibration signal with the zero-stress reference vibration signal and the dangerous stress reference vibration signal. The dangerous stress reference vibration signal can be obtained through calibration acquisition under a preset overload level, thereby concretizing the dangerous state into a comparable vibration template. When the current signal has a high degree of consistency with the dangerous stress template, the system can output rejection and lock commands, so that the high-risk carrier is intercepted before entering the subsequent link; when the current signal does not conform to the dangerous template and deviates from the zero-stress template, the system outputs an alarm or lock command to handle the uncertain state, thereby realizing non-contact, repeatable, and workstation-based detection and control of overload stress anomalies. Attached Figure Description

[0021] Figure 1 This is the overall flowchart of the present invention; Figure 2 This is an overall structural diagram of the present invention; Figure 3 This is a schematic diagram of the dynamic inertia fingerprint detection of the present invention; Figure 4 This is a flowchart of the deviation calculation for dynamic inertia fingerprint in this invention; Figure 5 This is a schematic diagram of the stress anomaly detection method of the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will now be described with reference to the accompanying drawings.

[0023] like Figure 1As shown in Figure 2, the method of the present invention is used for self-service management of intelligent carriers in classified management scenarios. The intelligent carrier can be a classified document storage box, a classified media storage box, a storage carrier with a closure component, or other classified carrier units that can be repeatedly transferred. The system can be composed of a sensing and acquisition end, an edge processing end, a central control end, and a data storage end. The sensing and acquisition end includes an appearance imaging device, an RFID reading and writing device, and a positioning device. Some workstations can also be equipped with an automated transmission mechanism, a pneumatic excitation device, and a non-contact laser displacement sensor or a Doppler vibration meter. The central control end is used for multimodal fusion recognition, identity verification, status decision-making, and digital twin model maintenance. The data storage end is used to store the carrier's digital fingerprint and digital twin model.

[0024] In one embodiment, the site of the classified area is arranged according to the following workstations: warehousing station, outbound station, handover station, and inspection station. Each warehousing, outbound, or handover station is equipped with an appearance imaging device and an RFID reader / writer, and outputs carrier positioning data through a positioning device. The appearance imaging device can be an industrial camera or a structured light camera, with a resolution of 1920×1080 or higher, a frame rate of 15fps to 60fps, and exposure time configured via automatic exposure or a preset exposure table to adapt to changes in lighting and reflections. The RFID reader / writer can be a UHF or HF reader / writer, reading the RFID tag code of the smart carrier to obtain carrier metadata. The carrier metadata may include at least the carrier model, batch number, volume specifications, type of closure component, and allowable loading range. The positioning device can use visual positioning, geomagnetic positioning, or RFID access control-based area positioning. The positioning data can be represented by two-dimensional coordinates and a timestamp, or by an area code and a timestamp, to meet different deployment cost and accuracy requirements. The appearance image data, RFID tag codes, and location data are all timestamped by the edge processing terminal and reported to the central control terminal for subsequent unified alignment and integration.

[0025] In one embodiment, the carrier digital fingerprint is used to uniformly represent multi-source information collected from the same carrier at different workstations and at different times, facilitating subsequent consistency comparison and tracking. The carrier digital fingerprint includes at least appearance features, carrier metadata, and location data. Appearance features can be obtained in several ways. The first approach uses traditional feature descriptors, extracting ORB, SIFT, or SURF feature points from the appearance image after denoising and distortion correction to form a feature descriptor vector. The second approach uses a deep learning feature extraction network, inputting the appearance image into a convolutional neural network to obtain a fixed-dimensional feature vector. The network can be ResNet18, MobileNetV3, or other lightweight networks, and the feature vector dimension can be 128, 256, or 512. The third approach uses object detection and local feature stitching, first using a detection network to locate parts such as the carrier nameplate, closed components, and handles, then extracting features from the local areas and stitching them together to form appearance features. To ensure consistency across workstations, white balance correction and histogram equalization can be performed on the image before appearance feature extraction, and feature drift caused by posture changes can be reduced by fixing the shooting angle or setting a positioning level at each workstation. Appearance features, as part of the carrier's digital fingerprint, can provide supplementary constraints when RFID is misread or the location drifts. The purpose is to avoid misjudging the carrier's identity when a single source of information fails in a local condition.

[0026] In one embodiment, the digital twin model is used to carry the full life cycle state and event trajectory of the intelligent carrier. Its function is not simply to store data, but to unify the carrier's digital fingerprint, state machine, event log and rule engine, so that the judgment result of each entry, exit or handover can be written and used for subsequent verification and linkage.

[0027] When constructing a digital twin model, a twin instance is created for each carrier using the RFID tag code as the primary key. Each twin instance contains at least static attributes, dynamic states, and event sequences. Static attributes correspond to carrier metadata. Dynamic states include at least the current location or region, the most recently confirmed carrier identity, the most recent carrier state, the state of closed components, the most recent operator identity verification result, and the most recently output permission, lock, or alarm command. Event sequences are used to append timestamps to store summaries of appearance features collected at each workstation, RFID read records, location records, multimodal fusion identification results, dynamic inertia fingerprint detection results, and stress anomaly detection results. To facilitate engineering implementation, the digital twin model can be implemented as a combination of relational database tables and time-series database tables. The relational database stores static attributes and current states, while the time-series database stores event sequences and sensor time-series data indexes. The central control terminal creates, updates, and queries twin instances via API. To avoid mis-merging caused by inconsistent timestamps from different data sources, the central control unit can align appearance image data, RFID tag codes, and location data according to timestamp windows. The window length can be selected from 0.5s to 3s, and adjusted according to the workstation cycle and network latency.

[0028] In one embodiment, identity verification is performed before performing inbound, outbound, or handover operations. Only after successful identity verification is control operation on the smart carrier permitted. This aims to bind operational permissions to specific actions, reducing impersonation and unauthorized access. Facial biometric data can be collected by a workstation camera at a distance of 1m to 2m to capture a frontal image and extract facial feature vectors. The feature vector dimension can be 128 or 512. The facial template library can be imported from the existing access control system or centrally registered when the system goes live. The template update cycle can be 30 days to 180 days to adapt to changes in personnel appearance. RFID badge identification codes can be read using a short-range reader at a distance of 0.02m to 0.2m to reduce cross-reading.

[0029] One approach to identity verification is to compare the facial feature vector with a person's facial template using cosine similarity. The similarity threshold can be selected from 0.4 to 0.7 and determined by balancing the false acceptance rate and the rejection rate. Simultaneously, the RFID badge code is matched against the personnel badge access control table to obtain the badge matching result. If both pass, the identity verification is considered successful. Another approach is to use facial comparison as a strong verification and badge matching as a weak verification. Passing is achieved when the facial verification passes and the badges belong to the same work group or work area. A further approach is to replace facial verification with vein or iris verification in nighttime or unstable lighting conditions, while maintaining the matching process with the badge access control table to ensure the traceability of personnel identity and permissions.

[0030] In one embodiment, after entering the warehousing, warehousing, or handover process for classified carriers, multimodal fusion identification is performed to determine the carrier's identity and status, and an access permission command, a lock command, or an alarm command is output based on the carrier's status. After receiving the appearance image data, RFID tag code, and location data, the central control terminal first calculates the appearance recognition confidence level, RFID recognition confidence level, and location consistency confidence level, respectively.

[0031] One way to achieve confidence in appearance recognition is to calculate the cosine similarity between the current appearance feature and the most recently confirmed appearance feature vector in the digital twin model and map it to 0 to 1; another way is to input the appearance feature vector into a lightweight classifier and output the probability of belonging to the RFID tag code. The classifier can be a logistic regression, support vector machine or small multilayer perceptron.

[0032] One way to implement RFID identification confidence is to calculate it based on whether the read RFID tag code is consistent with the expected target of the current workstation and the consistency of the number of reads. The number of reads can be 3 to 10, and the confidence is determined by majority vote. Another way is to calculate it based on RSSI stability and read success rate.

[0033] One way to achieve the confidence level of positioning consistency is to determine whether the positioning data falls within the allowable space range of the workstation and output 0 or 1; another way is to output continuous values ​​based on the positioning error model. The positioning error model can be obtained through UWB calibration, and the standard deviation of the error can be in the range of 0.1m to 0.5m.

[0034] The three factors are then weighted and summed according to preset weighting coefficients to obtain the fusion confidence score. These weighting coefficients can be optimized through historical data playback when the system goes live. Common settings are 0.3 to 0.6 for appearance, 0.3 to 0.6 for RFID, and 0.1 to 0.4 for positioning. Specific values ​​are adjusted based on on-site obstruction and RFID read / write stability. The fusion threshold can be selected from 0.6 to 0.9; a higher threshold is more conservative, making it easier to trigger locking or alarms but lowering the risk of false release. The carrier status can include at least normal, pending verification, boundary violation, and inconsistent identity. The central control terminal determines the carrier status and outputs corresponding instructions based on the fusion confidence score, positioning area, and historical status transition rules.

[0035] The "allow access" command can be used to allow access control, continue the operation of the conveyor line, or continue the handover process. The "lock" command can be used to stop the conveyor line, lock the carrier, or freeze the carrier's subsequent operations in the system. The "alarm" command can be used for audible and visual alarms or to push notifications to the management terminal. Finally, the identity verification result, carrier identity, carrier status, and corresponding commands are written into the digital twin model, enabling subsequent processes to query and perform consistency verification.

[0036] In a further embodiment, dynamic inertia fingerprint detection is introduced to prevent swapping of items of equal weight.

[0037] like Figure 3 As shown, at the workstation equipped with an automated transfer mechanism, which can be a robotic arm, a gripper-type handling unit, the drive section of a roller conveyor, or a rotary table, the automated transfer mechanism is controlled to grasp or carry the intelligent carrier and perform non-uniform speed testing actions. Non-uniform speed testing actions include at least acceleration, deceleration, or rotational maneuvers, the purpose of which is to induce significant angular or linear acceleration in the carrier during controlled motion, making the influence of internal mass distribution on the dynamic response observable. One example of a test action is accelerating from 0 to 0.6 m / s in 0.5 s, holding for 0.5 s, and then decelerating to 0 in 0.5 s; another example is a rotary table accelerating from 0 to 120 deg / s in 1 s and then decelerating to 0 in 1 s. The action parameters can be determined according to the carrier size and safety constraints; acceleration can be selected from 0.2 m / s² to 2.0 m / s², and angular acceleration can be selected from 30 deg / s² to 300 deg / s², ensuring no slippage or structural damage occurs.

[0038] During the test, real-time current load data and position feedback data of the drive motor are collected, and a dynamic inertia fingerprint is constructed based on these two types of data. To clearly define the data, the real-time current load data can be a sampling sequence of the motor phase current, q-axis current, or bus current output by the servo driver, with a sampling frequency ranging from 200Hz to 5000Hz. The position feedback data can be a sequence of angular position from the motor encoder or a sequence of angular velocity output by the driver, with a sampling frequency consistent with or an integer multiple thereof of the current sampling frequency. The reason for choosing current and position feedback data is that there is a correspondence between the electromagnetic torque output by the motor in the servo drive system and the current. The position feedback data can be used to derive velocity and acceleration, and the equivalent inertia of the carrier and its internal loads will affect the magnitude of the torque required and the actual acceleration response under the same motion control command.

[0039] In engineering implementation, it's not necessary to explicitly solve the complete dynamic model. Instead, the current sequence and acceleration-related sequence can be combined into an inertia feature vector, forming a comparable fingerprint. One feasible approach is to differentiate the position feedback data numerically to obtain angular velocity, and then differentiate it again to obtain angular acceleration. The current sequence is converted into a torque-related sequence based on the motor torque constant. Then, statistics for the torque-related sequence and angular acceleration sequence are calculated for acceleration, constant speed, and deceleration phases, and concatenated into an inertia feature vector. These statistics can include mean, variance, peak value, rise time, and area under the integral within the segment. Another approach is to time-align the torque-related sequence and angular velocity sequence, directly take a fixed-length sampling segment, normalize it, and use principal component analysis to compress the segment to 64 or 128 dimensions to form the inertia feature vector. Another approach involves inputting current and position feedback data into a lightweight one-dimensional convolutional network, whose output embedding vector serves as the inertia feature vector. This network can contain 3 to 6 layers of one-dimensional convolutions and global average pooling, with an embedding dimension of 32 to 256. The network training data comes from sampling sequences of historical normal carriers, and the loss function can employ a contrastive learning loss based on the proximity of samples with the same RFID tag and the greater distance between samples from different carriers. Regardless of the implementation method, the inertia feature vector is ultimately written into the digital twin model as a dynamic inertia fingerprint for reuse in subsequent stages.

[0040] In a further embodiment, such as Figure 4 As shown, the system calculates the deviation from the kinetic inertia fingerprint and outputs control commands. The system retrieves the standard inertia model associated with the RFID tag code or the original kinetic inertia fingerprint from historical warehousing. The standard inertia model can be a set of inertia feature vectors pre-collected under standard loading conditions for this type of carrier, or a set of inertia feature vectors corresponding to different loading levels, which can be divided according to the maximum allowable loading values ​​of 0.25, 0.50, 0.75, and 1.00. The original kinetic inertia fingerprint from historical warehousing can be the inertia feature vectors recorded during the carrier's 1st to 5th most recent warehousing. The current kinetic inertia fingerprint is then compared with the standard inertia model or the original kinetic inertia fingerprint to obtain the deviation value, calculated using Euclidean distance.

[0041] To enhance robustness, the inertia feature vector can be normalized before comparison to make it insensitive to motor voltage fluctuations and ambient temperature changes. The inertia threshold can be determined through pre-launch experiments. During the experiment, multiple batches of samples are collected under normal loading conditions to calculate the Euclidean distance distribution. The 95th or 99th percentile is taken as the initial value of the inertia threshold, and then fine-tuned according to the on-site false locking rate. The common range of inertia threshold values ​​is related to the feature dimension. When the feature dimension is 64 and has been normalized, the inertia threshold can be selected in the range of 0.3 to 1.2. When the deviation value is not greater than the inertia threshold, warehousing, outbound, or handover operations are allowed. When the deviation value is greater than the inertia threshold, an alarm command or locking command is output, and the result is written into the digital twin model for subsequent verification or traceability.

[0042] In a further embodiment, a non-contact modal detection step is introduced to detect stress anomalies caused by overload inside the intelligent carrier. For example... Figure 5 As shown, after the intelligent carrier is transported to the detection station, it is first confirmed to be in a stationary state. The stationary state can be confirmed by the conveyor line stop signal, the positioning data changing by less than 0.05m within 1 second, or the background noise of the vibration meter being lower than the threshold. The confirmation that the physical locking mechanism of the closed component is in the closed position can be achieved by the latch position switch, Hall sensor, RFID latch tag, or locking posture recognition in the appearance image.

[0043] After confirmation, the pneumatic excitation device is controlled to emit calibration fluid pulses to the surface of the closed component as a transient excitation source. The fluid can be air or inert gas, the preset pressure can be selected from 0.05MPa to 0.30MPa, and the preset duration can be selected from 5ms to 80ms. To ensure excitation consistency, a pressure sensor can be used to control the pulse waveform in a closed loop and record the actual pulse parameters into the digital twin model.

[0044] Subsequently, a non-contact laser displacement sensor or Doppler vibration meter is used to simultaneously acquire the micro-displacement time-domain vibration signal of the closed component after impact. The micro-displacement time-domain vibration signal is a displacement or velocity sequence that varies with time, commonly in the form of a displacement-time curve, with units in μm. The sampling frequency can be selected from 2000Hz to 50000Hz, and the sampling duration can be selected from 0.2s to 2.0s to ensure that the main attenuation process is included. Non-contact acquisition is chosen to avoid wear or contamination of the closed component surface by contact probes, and also to facilitate rapid detection within the management cycle.

[0045] In a further embodiment, the system acquires zero-stress reference vibration signals and critical stress reference vibration signals from the same type of intelligent carrier. The zero-stress reference vibration signal can be acquired from the same type of intelligent carrier under preset no-load conditions, defined as the closed assembly locked in place and the internal load mass being 0 or less than 0.05 of the maximum permissible load. The critical stress reference vibration signal is acquired from the same type of intelligent carrier under a preset overload level, defined as 1.10 to 1.50 of the maximum permissible load, and can be determined in conjunction with the experimental boundary of the risk of the closed assembly breaking open. The acquisition of the reference signals should maintain consistent calibration fluid pulse parameters and installation posture with the on-site testing, and at least 30 to 200 samples should be collected for each model to cover manufacturing differences and environmental variations. Subsequently, the current micro-displacement time-domain vibration signal is compared with the zero-stress reference vibration signal to obtain the zero-stress consistency degree, and the current micro-displacement time-domain vibration signal is compared with the critical stress reference vibration signal to obtain the critical consistency degree.

[0046] Furthermore, consistency comparison can be implemented in several ways. The first method involves time alignment and amplitude normalization of the signals, followed by calculating the normalized cross-correlation maximum value; the closer the cross-correlation maximum value is to 1, the more similar the signals. The second method calculates the mean square error and maps it to a consistency score between 0 and 1; the smaller the error, the higher the consistency score. The third method uses dynamic time warping distance to measure waveform similarity and maps it to a consistency score, adapting to minor time scaling differences. To avoid the influence of background noise, bandpass filtering can be applied to the signal; the passband can be selected from 20Hz to 5000Hz, adjusted according to the material and size of the closed-loop components.

[0047] In a further embodiment, commands are output based on the hazard consistency degree, zero-stress consistency degree, and a threshold. When the hazard consistency degree is not less than a preset hazard threshold, a high internal pressure overload state is determined, and a rejection command and a lock command are output. When the hazard consistency degree is less than the hazard threshold and the zero-stress consistency degree is not less than a preset baseline threshold, a passage-allowed state is determined, and a passage-allowed command is output. When the hazard consistency degree is less than the hazard threshold and the zero-stress consistency degree is less than the baseline threshold, an alarm command or a lock command is output. The hazard threshold and the baseline threshold can be determined through experimental statistics. During the experiment, the consistency degree distribution is calculated for zero-stress samples and hazard stress samples respectively, and a threshold combination is selected that makes the detection rate of hazard samples reach above 0.95 and the false alarm rate of zero stress below 0.05. If the normalized cross-correlation maximum value is used as the consistency degree, the hazard threshold can be selected from 0.80 to 0.95, and the baseline threshold can be selected from 0.70 to 0.90; the specific values ​​can be adjusted according to the material stiffness, closed component structure, and sensor noise level. The detection results and corresponding commands are also written into the digital twin model for easy verification and traceability in subsequent stages.

[0048] In a further embodiment, the carrier digital fingerprint also includes feature vectors of appearance characteristics, and the digital twin model stores the timestamps of the feature vectors and positioning data. This is done to support cross-time appearance consistency verification and trajectory consistency verification. For example, when RFID reading / writing is interfered with by metal shelves, the historical sequence of appearance feature vectors can still be used to determine whether it is the same carrier. The feature vector can be updated every time it enters, leaves, or is handed over, or it can be updated only when the fusion confidence level is higher than 0.90, to prevent misjudgments from contaminating the twin model. The timestamp can use millisecond-level Unix timestamps or ISO time formats and be stored together with the workstation code for easy retrieval.

[0049] Through the above implementation methods, the present invention achieves full-process consistency control based on carrier digital fingerprint and digital twin model without changing the basic process of classified carrier management. It also introduces dynamic inertia fingerprint detection and non-contact modal detection at key work stations, so that equal weight swapping and overload stress anomalies can be identified and trigger corresponding control commands within the management cycle.

[0050] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal recognition-based self-service management method for intelligent carriers, used for self-service management of intelligent carriers in classified management scenarios, characterized in that, include: The appearance image data of the smart carrier is collected and appearance features are extracted. The RFID tag code of the smart carrier is read to obtain carrier metadata. The positioning data of the smart carrier is collected. The appearance features, the carrier metadata and the positioning data are combined to form a digital fingerprint of the carrier, and a digital twin model associated with the RFID tag code is constructed accordingly. Before performing warehousing, warehousing, or handover operations on the smart carrier, the operator's facial biometric features are collected and the operator's RFID badge code is read to perform identity verification. After the identity verification is passed, control operations on the smart carrier are allowed to be initiated. In the process of managing classified carriers during warehousing, warehousing, or handover, the appearance image data, RFID identification code, and positioning data of the smart carrier are collected. Multimodal fusion identification is performed to determine the carrier identity and carrier status of the smart carrier, and an access permission command, a lock command, or an alarm command is output based on the carrier status. The identity verification result, the carrier identity, the carrier status, and the permission instruction, the lock instruction, or the alarm instruction are written into the digital twin model.

2. The method according to claim 1, characterized in that, The carrier digital fingerprint also includes a dynamic inertia fingerprint for preventing swapping of equal-weight carriers, and the classified carrier control process further includes: controlling an automated transmission mechanism to grab or carry the intelligent carrier, and executing non-uniform speed test actions according to preset motion control commands, the test actions including acceleration, deceleration or rotational maneuvers; during the execution of the test actions, collecting real-time current load data and position feedback data of the drive motor of the automated transmission mechanism, and constructing the dynamic inertia fingerprint based on the real-time current load data and the position feedback data.

3. The method according to claim 2, characterized in that, Also includes: Retrieve the standard inertia model or the original dynamic inertia fingerprint associated with the RFID tag code; compare the dynamic inertia fingerprint with the standard inertia model or the original dynamic inertia fingerprint to obtain a deviation value, the deviation value being the Euclidean distance between the two inertia feature vectors; when the deviation value is not greater than a preset inertia threshold, allow the smart carrier to perform the entry, exit, or handover operation; when the deviation value is greater than the preset inertia threshold, output the alarm command or the locking command.

4. The method according to claim 1, characterized in that, The classified carrier control process also includes a non-contact modal detection step for detecting stress anomalies caused by overload inside the intelligent carrier. The non-contact modal detection step includes: when the intelligent carrier is transferred to the detection station, confirming that the intelligent carrier is in a stationary state and that the physical locking mechanism of its closed component is in the closed position; controlling a pneumatic excitation device to emit a calibration fluid pulse with a preset pressure and preset duration as a transient excitation source to the surface of the closed component; and using a non-contact laser displacement sensor or Doppler vibration meter to synchronously collect the micro-displacement time-domain vibration signal of the closed component after being impacted by the calibration fluid pulse.

5. The method according to claim 4, characterized in that, Also includes: The zero-stress reference vibration signal and the critical stress reference vibration signal of the same type of intelligent carrier are acquired. The critical stress reference vibration signal is obtained by performing the calibration fluid pulse excitation on the same type of intelligent carrier under a preset overload level and acquiring it by the non-contact laser displacement sensor or the Doppler vibration meter. The consistency of the micro-displacement time-domain vibration signal and the zero-stress reference vibration signal is compared to obtain the zero-stress consistency degree, and the consistency of the micro-displacement time-domain vibration signal and the critical stress reference vibration signal is compared to obtain the critical consistency degree.

6. The method according to claim 5, characterized in that, When the hazard consistency is not less than a preset hazard threshold, the intelligent carrier is determined to be in a high internal pressure overload state and a rejection command and a lock command are output; when the hazard consistency is less than the preset hazard threshold and the zero stress consistency is not less than a preset benchmark threshold, the intelligent carrier is determined to be in a passable state and the passable command is output; when the hazard consistency is less than the preset hazard threshold and the zero stress consistency is less than the preset benchmark threshold, the alarm command or the lock command is output.

7. The method according to claim 1, characterized in that, The carrier digital fingerprint also includes a feature vector of the appearance features, which is obtained by performing feature extraction on the appearance image data, and the digital twin model stores the feature vector and the timestamp of the positioning data.

8. The method according to claim 1, characterized in that, The identity verification includes: comparing the facial biometric features with a pre-registered personnel facial template to obtain a facial comparison result, and matching the RFID work badge identifier code with a pre-registered personnel work badge permission table to obtain a work badge matching result. When the facial comparison result is passed and the work badge matching result is passed, the identity verification is determined to be successful.

9. The method according to claim 1, characterized in that, The multimodal fusion identification includes: calculating the confidence scores for appearance recognition, RFID recognition, and positioning consistency, respectively, and then weighting and summing these scores according to preset weighting coefficients to obtain a fusion confidence score; outputting the carrier identity and carrier status when the fusion confidence score is not less than a preset fusion threshold, and outputting the locking command or the alarm command when the fusion confidence score is less than the preset fusion threshold.

Images