A method for identifying container numbers in tally

Abstract

The present invention discloses a method for identifying container number tallying, which relates to the technical field of intelligent container tallying. The key technical points are: deploying multimodal sensing devices and performing initial calibration, pre-training an improved YOLOv7-tiny target detection model, a Transformer-based cGAN character segmentation model, and a hybrid recognition model integrating a lightweight CNN-Transformer-capsule network; through the collaborative multimodal data acquisition of multimodal sensing devices, the system can effectively integrate the texture information of RGB images, the spatial structure of depth images, and the three-dimensional coordinates of point cloud data, solving the problem that traditional single visual recognition is affected by lighting, occlusion, etc. The self-calibration mechanism corrects the sensor's spatiotemporal deviation in real time, enabling the system to accurately extract container number features even in complex scenarios such as strong light reflection, low illumination at night, and character wear.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent container tallying, and in particular to a method for identifying container numbers for tallying. Background Art

[0002] In the container operation scenarios of modern ports and logistics centers, accurate identification of container numbers is the core link of tallying work, which is directly related to the efficiency, safety and traceability of cargo transportation. Traditional manual identification methods rely on tally clerks to check each box one by one. During busy peak operation periods, not only is the work intensity extremely high, but it is also very easy to cause omissions due to fatigue. In actual operations, manual identification often has problems such as character recording errors and number confusion, resulting in a disordered order of cargo loading and unloading. In serious cases, it may even cause chaos in transportation scheduling, resulting in the wrong flow of cargo, which brings great uncertainty and risk to logistics transportation.

[0003] Although existing automatic recognition technology has improved efficiency to a certain extent, it still has many limitations. The recognition method based on single vision is often affected by environmental factors in actual port operations. For example, at noon in summer, strong sunlight directly hits the surface of the container, and the resulting reflection causes the character area to overexpose, making it difficult for the camera to capture a clear image; at night or in dimly lit warehouses, the images taken in low-light environments have many noise points and the character features are blurred. These situations greatly reduce the recognition accuracy. In actual applications, many logistics parks use single-camera recognition systems, and recognition errors caused by lighting problems frequently occur, which in turn causes the wrong shipment of goods, seriously affecting customer satisfaction and corporate reputation.

[0004] Although multi-sensor fusion recognition schemes introduce depth information to assist in identification, in practical applications, the immaturity of data fusion algorithms makes it difficult to effectively integrate the data collected by various sensors. For example, in automated terminals, even if the equipment is equipped with both visual cameras and lidar, due to low data fusion efficiency and insufficient feature extraction, processing a single container number still takes a long time, which cannot meet the needs of efficient operations during peak hours in ports. In addition, existing technologies lack a dynamic verification mechanism. Faced with common abnormalities such as stains on the surface of containers, characters worn out due to long-term use, and partial occlusion caused by stacking, accurate recognition is difficult and often requires manual secondary verification, which greatly increases operating costs and time costs. To this end, we provide a method for container number tallying recognition to address the above technical problems. Summary of the Invention

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] A method for identifying container number tally includes the following steps:

[0007] S1. System initialization and parameter configuration: deploy distributed stereo vision array, millimeter wave radar and lidar and perform initial calibration, configure reinforcement learning parameters, pre-train the improved YOLOv7-tiny target detection model, the Transformer-based cGAN character segmentation model and the hybrid recognition model integrated with lightweight CNN-Transformer-capsule network, build the character structure knowledge graph and initialize the dynamic Bayesian network parameters.

[0008] The initial calibration includes: calibrating the internal and external parameters of the distributed stereo vision array using Zhang's calibration method. The internal parameter calibration accuracy reaches 0.1 pixel level, and the external parameter calibration error is controlled within ±0.5°. The pre-training model adopts a multimodal pre-training strategy based on contrastive learning. The generalization ability of the model is enhanced by constructing an image-point cloud-text contrastive learning task. The temperature parameter of the contrastive learning is set to 0.07, and the positive and negative sample ratio is 1:3.

[0009] S2. Collaborative multimodal data acquisition: A distributed stereo vision array with a self-calibrating mechanism, combined with multi-sensor fusion positioning of millimeter-wave radar and lidar, simultaneously collects RGB images, time-of-flight depth images, millimeter-wave point cloud data, and lidar point cloud data from multiple sides of the container. A dynamic threshold triggering and adaptive exposure control strategy based on a deep Q network is used to acquire multimodal data including the container number area.

[0010] The dynamic threshold triggering strategy uses a two-layer Q network architecture to achieve multi-objective optimization, including the target network and the real network. The target network parameters are updated every 100 steps, and the real network uses the Adam optimizer with a learning rate of 0.001. The adaptive exposure control strategy dynamically adjusts the exposure time by calculating the image entropy value. The exposure time range is 1ms-100ms, and the adjustment step size is 0.1ms.

[0011] S3. Heterogeneous Data Fusion Preprocessing: A cross-modal feature fusion network based on the Transformer architecture, combined with a dynamic routing mechanism, is used to fuse multi-source data at the feature level. A noise reduction model based on a generative adversarial network and a deep learning-based geometric distortion correction algorithm are used to preprocess the fused data.

[0012] The cross-modal feature fusion network includes an 8-head multi-head self-attention mechanism, and the dynamic routing mechanism adaptively assigns fusion weights by calculating the quantum entanglement similarity of features from different modalities. The similarity is calculated using the quantum state inner product formula. The denoising model consists of a generator and a discriminator. The generator adopts a U-Net structure, consisting of 5 downsampling blocks and 5 upsampling blocks, and the discriminator adopts a PatchGAN structure. The loss function of adversarial training is LSGAN.

[0013] S4, Intelligent Character Region Localization: Based on the improved YOLOv7-tiny object detection model, integrating the spatiotemporal attention mechanism, character structure knowledge graph, and semantic association analysis based on graph neural networks, character region detection is performed on preprocessed data, and the localization results including bounding box coordinates, confidence level, character category probability, and semantic association score are output;

[0014] The improved YOLOv7-tiny model uses a collaborative optimization architecture combining progressive knowledge distillation and dynamic network pruning. The knowledge distillation temperature parameter is set to 4, and the pruning ratio is 30%. This reduces the number of model parameters by 40%. The spatiotemporal attention mechanism captures the changing characteristics of character regions at different time steps through a spatiotemporal feature pyramid structure. The feature pyramid contains three scales, corresponding to 1 / 4, 1 / 8, and 1 / 16 resolutions of the original image.

[0015] S5. Adaptive Character Segmentation and Normalization: Utilizes the Transformer-based cGAN character segmentation algorithm, combined with morphological post-processing for topological structure analysis; and achieves accurate segmentation and normalization of character images through an adaptive normalization network based on a differentiable deformation module.

[0016] The differentiable deformation module uses a hybrid deformation model that combines bilinear interpolation with thin plate spline transformation. The morphological post-processing of topological structure analysis includes calculating topological invariants such as the Euler number and the number of holes in the connected area of ​​the characters.

[0017] S6, Multi-model Collaborative Character Recognition: This uses a hybrid model integrating lightweight CNN, Transformer, and capsule network, combined with bidirectional knowledge distillation technology and dynamic weight fusion strategy to perform character classification, output character probability distribution, and combine them in spatial order to form preliminary recognition results;

[0018] Among them, the bidirectional knowledge distillation adopts an adversarial knowledge transfer mechanism. The teacher model guides the student model learning through soft labels. The student model optimizes the feature expression of the teacher model through adversarial training feedback. The discriminator of adversarial training adopts a multi-layer perceptron, which contains 2 hidden layers and 256 neurons in each layer. The dynamic weight fusion strategy adopts a reinforcement learning optimization framework to dynamically adjust the model weights based on recognition accuracy and computational efficiency as optimization goals. The reward function of reinforcement learning is recognition accuracy × 0.8 + computational speed × 0.2.

[0019] S7. Multi-dimensional Intelligent Verification and Decision-making: Build a verification and decision-making model based on a dynamic Bayesian network. This model combines format compliance verification, checksum correctness verification, spatiotemporal consistency verification, multimodal data consistency verification, and trend prediction verification based on historical data to assess the credibility of container number recognition results. When a verification result is abnormal, a targeted verification enhancement strategy is initiated.

[0020] The dynamic Bayesian network uses a reasoning mechanism that integrates evidence theory and Bayesian optimization. The basic probability distribution function of evidence theory is obtained through historical data statistics, and the acquisition function of Bayesian optimization is the expected improvement (EI) function. The targeted verification enhancement strategy adopts a meta-verification framework to automatically select the optimal verification enhancement scheme based on the verification anomaly type. The meta-verification framework includes five verification enhancement strategies and optimizes the strategy selection probability through a genetic algorithm.

[0021] S8. Result Output and Blockchain Evidence Storage: The verified container number and related information are output to the tally system. At the same time, blockchain evidence storage technology based on zero-knowledge proof and combined with homomorphic encryption data processing methods are used to achieve tamper-proof evidence storage.

[0022] Among them, zero-knowledge proof uses zk-STARKs technology, the generated proof length is 1024 bits, and the verification time is less than 100ms; homomorphic encryption uses the BGV homomorphic encryption scheme, which supports addition and multiplication operations on ciphertext, and the encryption parameter is set to modulus q=2 40 , the plaintext space is Z_2 10 ;Blockchain evidence storage uses side chain technology to achieve high-frequency transaction processing.

[0023] Furthermore, in S1, the character structure knowledge graph predefines constraints such as character spacing of 1.2-1.5 times the character height and aspect ratio of 0.8-1.2 to filter candidate boxes that do not meet the structural rules; the nodes of the knowledge graph include character type, character size, character spacing, etc., and the edges include spatial relationships, semantic relationships, etc. The knowledge graph is constructed using the Neo4j graph database, and the attributes of nodes and edges are determined by a combination of expert annotation and machine learning.

[0024] Furthermore, in S2, the self-calibration mechanism uses the checkerboard calibration board and IMU deployed in the transport channel to perform real-time spatiotemporal calibration of the stereo vision array, millimeter-wave radar, and lidar based on the Kalman filter algorithm. The state transfer matrix of the Kalman filter is determined by the motion model of the IMU, and the observation matrix is ​​determined by the corner point detection results of the checkerboard. The distributed stereo vision array adopts a non-uniform redundancy design and dynamically adjusts the sensor layout density according to the recognition requirements of the key areas of the container. The sensor density in the key areas is twice that of the non-key areas.

[0025] Furthermore, in S3, the geometric distortion correction algorithm adopts a differentiable mapping model based on differential homeomorphism, optimizes the mapping parameters by minimizing the image reprojection error, and the threshold of the reprojection error is set to 1 pixel; the generator and discriminator of the denoising model both use the LeakyReLU activation function with a slope of 0.2, and the momentum parameter of the batch normalization layer is set to 0.9.

[0026] Furthermore, in S4, the semantic association analysis of the graph neural network adopts a graph convolutional network (GCN), which contains 2 graph convolution layers, each with an output dimension of 128 and an activation function of ReLU; the dynamic update of the character structure knowledge graph is achieved through incremental learning. After each recognition result is updated, the confidence score of the knowledge graph is recalculated, and nodes and edges with a confidence score lower than 0.5 are deleted.

[0027] Furthermore, in S5, the Transformer-based cGAN character segmentation algorithm introduces an adaptive receptive field module in the generator. The receptive field size is dynamically adjusted according to the different morphological features of the characters, and the adjustment range is 3×3 to 11×11; the adaptive normalization network adopts an attention-guided spatial transformation network, and the attention weight map is generated by the Softmax function. The parameters of the spatial transformation network are learned through back propagation.

[0028] Furthermore, in S6, the lightweight CNN of the hybrid model adopts the MobileNetV3 architecture, the Transformer adopts a 6-layer encoder, the number of capsules in the capsule network is 16, and the capsule dimension is 32; the reinforcement learning optimization framework of the dynamic weight fusion strategy adopts the PPO algorithm, and the policy network and value network both use multi-layer perceptrons, the hidden layer size is 256, and the training batch size is 64.

[0029] Furthermore, in S7, format compliance verification is implemented through a probabilistic finite state automaton. The state transition probability of the automaton is obtained through historical data statistics, and the state transition threshold is set to 0.8; spatiotemporal consistency verification is constructed using a spatiotemporal graph convolutional network. The graph convolutional network contains 3 spatiotemporal convolutional layers, the time step is set to 5, and the spatial neighborhood size is 3×3.

[0030] Furthermore, in S8, the side chain of blockchain evidence storage adopts the practical Byzantine fault tolerance (PBFT) consensus mechanism, with the number of consensus nodes being 5 and the Byzantine node tolerance being 1; the evidence data is organized using a Merkle tree structure with a depth of 10. The leaf nodes store data hash values, and the parent nodes store a combination of child node hash values.

[0031] Furthermore, in S9, the construction of the dynamic training dataset adopts an active learning strategy, and the most valuable error cases are selected through uncertainty sampling. The uncertainty measurement adopts the predicted entropy value, and the entropy value threshold is set to 0.9; the reward function parameter optimization of reinforcement learning adopts the trust region policy optimization (TRPO) algorithm, the trust region size is set to 0.01, and the optimization step size is set to 0.5.

[0032] A system for container number tally identification method includes a hardware layer and a software layer; the hardware layer includes a multimodal perception unit, an edge computing unit, a trigger and control unit, and an auxiliary unit; the software layer includes a multimodal data processing module, an intelligent recognition module, an intelligent verification and decision module, an interface and evidence storage module, and a system management module. Among them, the edge computing unit adopts a heterogeneous computing architecture, integrating a GPU, an FPGA, an NPU, and a dedicated AI acceleration chip, and realizes intelligent allocation of computing tasks through a task scheduling optimization engine. The task scheduling optimization engine adopts a genetic algorithm with a population size of 50 and the number of iterations of 100.

[0033] A computer-readable storage medium stores a computer program, the program including a system initialization and parameter configuration module, a multimodal data collaborative acquisition control module, a heterogeneous data fusion preprocessing module, a character area intelligent positioning module, an adaptive character segmentation and normalization module, a multi-model collaborative character recognition module, a multi-dimensional intelligent verification and decision module, and a blockchain evidence interface module; the computer program, when executed by a processor, is used for the steps of a container number tally recognition method, wherein the multi-model collaborative character recognition module adopts a model integration uncertainty quantification method, estimates the uncertainty of the recognition result through the Monte Carlo Dropout technique, the Dropout probability is set to 0.5, and the number of Monte Carlo sampling is 10.

[0034] The present invention provides a method for identifying container numbers in tally, which has the following beneficial effects:

[0035] 1. Full-chain high-precision recognition, using a three-level cascade recognition architecture for precise processing. An improved YOLOv7-tiny model, combined with knowledge distillation and spatiotemporal attention, effectively addresses the issue of missed detection of small characters in high-speed motion. The Transformer-cGAN segmentation model achieves complete segmentation of corroded and adhered characters through dynamic receptive field and topological analysis. A lightweight integrated model fuses local texture, global semantics, and spatial hierarchical features, combined with a dynamic weighting strategy, to accurately distinguish easily confused characters, building a full-process high-precision processing system from target detection to semantic recognition.

[0036] 2. Environmental adaptive perception: The multimodal data acquisition module achieves environmental robustness through the collaboration of hardware and algorithms. The distributed vision array and IMU are jointly calibrated to quickly complete spatiotemporal calibration when the device vibrates, ensuring accurate registration of multi-source data. The dynamic threshold trigger strategy is combined with a two-layer Q network to adaptively adjust sensor parameters, maintaining stable data acquisition quality over a wide illumination range and in extreme weather conditions. Combined with GAN noise reduction and distortion correction technology, it forms an environmental adaptation closed loop from data acquisition to preprocessing.

[0037] 3. Multi-level verification decision-making and a three-level verification architecture build a data security protection system. The basic verification layer filters invalid data through format compliance verification and spatiotemporal trajectory analysis; the meta-verification framework of the enhanced verification layer integrates a multi-strategy dynamic response mechanism to trigger rapid review of low-confidence results; the blockchain evidence layer uses zero-knowledge proof and homomorphic encryption technology to achieve tamper-proof evidence of verification data. From format verification to exception handling to data evidence, the accuracy and traceability of tally data are fully guaranteed.

[0038] 4. The device-edge collaborative architecture adopts heterogeneous computing design at the hardware layer to achieve efficient processing. Cross-modal feature fusion and deep learning inference are completed on edge devices. The millisecond-level processing speed meets the port's high-speed throughput requirements of hundreds of boxes per hour. The edge computing architecture reduces network dependence and maintains stable operation in complex communication environments. It builds an intelligent tallying infrastructure that is both real-time and reliable, providing technical support for the automated operations of smart ports.

[0039] The organic combination of the above technical solutions enables the present invention to achieve a significant improvement in recognition performance in complex scenarios such as high speed, strong light, and rust, and builds a fully automated process from data collection to decision-making and evidence storage, providing a practical solution for the intelligent upgrade of port tallying operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 4 is a flow chart of the overall system of an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0042] Example: Figure 1 As shown in the figure, a container number tally recognition technology based on multimodal fusion and intelligent decision-making is disclosed. In the container number tally recognition system, the design of each model and algorithm is closely centered around the data processing and decision-making process. From multimodal data acquisition, feature extraction, character recognition, to result verification, each link has been carefully designed and optimized to achieve the goal of high-precision and high-robust recognition. The following will elaborate on the design process of each key model and algorithm, and provide specific examples based on actual application scenarios.

[0043] 1. Multimodal data acquisition:

[0044] 1.1, Self-calibration mechanism algorithm:

[0045] (1) Hardware deployment: A checkerboard calibration plate is installed at a fixed position in the container transport channel, and an IMU (inertial measurement unit) is integrated into the sensor device. The checkerboard calibration plate serves as a reference for visual calibration, and the IMU is used to monitor the motion status of the device in real time. The details are as follows:

[0046] Ten Xsens MTi-G-700 inertial measurement units (IMUs) are evenly installed at key locations on the gantry and sensor brackets of the container transport channel. Furthermore, twelve Basler ac A4112-8gm industrial cameras are installed in a matrix configuration on both sides and at the top of the channel. These cameras, paired with Schneider 12mm industrial lenses, have a distortion rate of <0.1%, enabling high-quality image capture. This constitutes a distributed stereo vision array capable of acquiring container parameters and images from multiple angles. The distributed stereo vision array uses Zhang's calibration method for both intrinsic and extrinsic calibration, achieving an intrinsic calibration accuracy of 0.1 pixel, an extrinsic calibration error of ±0.5°, and a translation error of ±0.8 cm.

[0047] The Distributed Stereo Vision Array (DSVA) is a stereo vision system based on a distributed deployment of multiple cameras (or visual sensors). By coordinating the perception data of multiple independent visual nodes, it achieves high-precision reconstruction of three-dimensional scenes, target positioning, and tracking. The DVA adopts a non-uniform redundancy design, deploying four industrial cameras per square meter in key areas where container numbers are located and two cameras per square meter in non-critical areas. This layout, tested through simulations of actual port scenarios, ensures recognition accuracy while reducing hardware costs. The self-calibration mechanism uses a checkerboard calibration plate every five meters in the transport channel. The IMU uses a high-precision inertial measurement unit (such as the XsensMTi-30). Using the Kalman filter algorithm, the stereo vision array, millimeter-wave radar, and lidar are calibrated in real time and space. The state transition matrix is ​​constructed using the acceleration and angular velocity data collected by the IMU, and the observation matrix is ​​constructed using the checkerboard corner detection results. This enables precise calibration between sensors and ensures the spatial consistency of multimodal data.

[0048] The IMU is installed in locations susceptible to vibration, such as the center of the gantry beam and the four corners of the sensor bracket, to ensure that the movement of the equipment can be fully perceived; industrial cameras are arranged at intervals of 2 meters on both sides of the channel, and one is installed every 3 meters on the top to form all-round visual coverage.

[0049] (2) Data acquisition: During the operation of the sensor, the IMU collects acceleration, angular velocity and other data at a frequency of 100 Hz, and the visual sensor regularly captures checkerboard images.

[0050] (3) State prediction: Based on the data collected by IMU, the state transition equation of the extended Kalman filter algorithm is used Predict the state of the sensor at the current moment.

[0051] Formula analysis: This equation is used to predict the sensor State of the moment ,in, It is the state transfer matrix that describes the motion law of the sensor. For example, when the onboard sensor moves with the vehicle, It can be constructed based on the vehicle kinematic model to convert the last moment Status Switch to the current moment; To control the input matrix, when the sensor has a motion command controlled by humans or the system hour, Indicates the effect of control input on the state; It is process noise, which is used to characterize interference factors that cannot be accurately modeled in actual motion, such as the bumps and vibrations of a vehicle when it is driving. It is usually assumed to obey a Gaussian distribution. , is the process noise covariance matrix, where It is a control quantity set by humans or the system, which is used to describe the active control or influencing factors of the outside world on the motion state of the sensor.

[0052] For example, when a container transport vehicle passes through a speed bump, the IMU collects acceleration mutation data as , combined with the preset and ,The equation is used to predict the state of the sensor after vibration, providing a reference for subsequent calibration.

[0053] (4) Observation update: By detecting the corner points of the chessboard image and combining the Zhang calibration method to calculate the external parameters of the sensor, the observation equation is constructed. , the predicted state is fused with the observation results, the spatiotemporal parameters of the sensor are updated, and self-calibration is achieved.

[0054] Formula analysis: This equation is used to establish the sensor prediction state Compared with actual observation data The relationship between is the observation matrix, which maps the state variables to the observation space, such as when visual calibration is performed through checkerboard images, According to the principle of Zhang's calibration method, the spatial posture parameters of the sensor can be converted into the coordinates of the checkerboard corner points in the image; To observe noise, it reflects the errors in actual observation, such as the measurement deviation caused by the resolution limitation of the camera image and light interference. It is usually assumed to obey Gaussian distribution. , is the observation noise covariance matrix.

[0055] For example, when a visual sensor captures a checkerboard image, the detected corner coordinates are used as , through the known and predicted status Calculate the observed value and the actual Compare and then update the sensor parameters.

[0056] Example: Take the container transport channel of the port as an example. When the transport vehicle passes through the speed bump, the strong vibration generated causes the visual sensor installed on the gantry to shift. At this time, the IMU collects the acceleration of the device in the X-axis direction at a frequency of 100Hz in real time. , the Y-axis angular velocity changes to Based on these data, the extended Kalman filter algorithm predicts the offset angle and displacement of the sensor in space. At the same time, the visual sensor captures the checkerboard image, detects the corner points and combines it with the Zhang calibration method to calculate the actual external parameter changes. The system fuses the predicted state with the actual observation results, adjusts the rotation and translation parameters of the sensor, and completes self-calibration within 2 seconds to ensure that the subsequently collected image data is accurate.

[0057] The Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters of the devices in the distributed stereo vision array. 20 sets of calibration images (resolution 1920×1080) are collected to calculate the intrinsic parameter matrix. , where the focal length parameter The accuracy reaches 0.1 pixel level. The focal lengths of the camera in the x and y axes (unit: pixel), respectively, reflect the zoom capability of the lens on the image, and the principal point coordinates The positioning error is less than 0.5 pixels.

[0058] 1.2. Timestamp synchronization mechanism:

[0059] Time calibration of multimodal sensing devices through a timestamp synchronization mechanism is the core link to ensure the spatiotemporal consistency of multi-source data. In its specific implementation, a combination of hardware timestamps and software algorithms is adopted: each device embeds a high-precision timestamp at the moment of data collection, and global clock synchronization is achieved through PTP or GPS timing. The edge computing unit monitors the time deviation of each device in real time, and dynamically adjusts the time offset based on the Kalman filter algorithm to control the time error within ±10ms.

[0060] The time calibration process includes: ① completing clock reference alignment during device initialization; ② periodically sending synchronization signals during operation (e.g., once per second); ③ carrying timestamps and marking the collection order during data collection. Through this mechanism, clock drift between multiple devices can be eliminated, and spatiotemporal misalignment of multimodal data caused by nanosecond time differences can be avoided. This ensures that RGB images, point cloud data, etc. have precise time correspondence during fusion, providing a reliable time reference for subsequent character positioning and multi-model collaborative recognition, and improving the accuracy of data fusion and system robustness in complex scenarios.

[0061] 1.3 Dynamic threshold triggering strategy algorithm:

[0062] The dynamic threshold triggering strategy uses a two-layer Q network architecture. The target network copies parameters from the real network every 100 steps. The real network uses the Adam optimizer with a learning rate of 0.001. The optimization objective function is set to "maximize detection accuracy × 0.7 + minimize false detection rate × 0.3". This weight distribution is determined through multiple simulation experiments and can effectively achieve multi-objective optimization. The adaptive exposure control strategy dynamically adjusts the exposure time based on the image entropy value. When the image entropy value is lower than the threshold of 80, the exposure time is increased in steps of 0.1ms; when it is higher than the threshold of 120, the exposure time is reduced to ensure that the images collected under different lighting conditions are clear and usable. The details are as follows:

[0063] (1) State definition: Determine the state space, including eight dimensions of environmental characteristics, such as the current frame image entropy (reflecting image clarity, ranging from 0 to 8), depth map variance (measuring depth data stability, ranging from 0 to 1000), and millimeter wave radar point cloud density (the number of point clouds per unit volume, ranging from 0 to 50 points / m³).

[0064] The trigger and control unit uses three Advantech UNO-3083 intelligent controllers, deployed in the channel monitoring room, equipment power box, and edge computing node. The multimodal perception unit includes 16 Basler industrial cameras, installed symmetrically in two rows on both sides of the channel. Eight Intel RealSense D455 TOF depth sensors are spaced between the cameras. Four Continental ARS408-21 millimeter-wave radars are installed above the channel entrances and exits. Two Velodyne VLP-16 lidars are placed high in the middle of the channel. The millimeter-wave radar and lidar are aligned using the NDT algorithm, with a registration error of less than 5 cm.

[0065] 100 sets of calibration data from the Continental ARS408-21 millimeter-wave radar and Velodyne VLS-128 lidar were simultaneously collected, and point cloud registration was performed based on the Normal Distributions Transform (NDT) algorithm.

[0066] By configuring the registration parameters with a grid resolution of 0.1m and 50 iterations, high-precision registration with a root mean square error of less than 5cm was achieved. The time synchronization module uses the IEEE 1588 precision clock protocol and achieves nanosecond-level synchronization of multiple sensors through a hardware timestamp counter, with synchronization accuracy controlled within ±10μs.

[0067] Industrial cameras are installed on both sides of the channel, 5 meters above the ground, with two rows of cameras spaced 1.5 meters apart; TOF depth sensors are installed alternately with cameras to ensure data complementarity; millimeter-wave radars are installed horizontally above the entrances and exits to cover the vehicle traffic area; and lidar is installed on an 8-meter-high column in the middle of the channel to obtain a wider scanning range.

[0068] (2) Network architecture: A two-layer deep Q network (DQN) architecture is adopted, which includes a target network and a real network. The target network parameters are copied and updated from the real network every 100 steps. The real network is trained using the Adam optimizer with a learning rate set to 0.001.

[0069] (3) Reward function design:

[0070] Designing the reward function , comprehensively considering the recognition accuracy, acquisition efficiency and false trigger rate, guide the algorithm to learn the optimal trigger strategy, where, The recognition accuracy is the ratio of the number of correctly identified container numbers to the total number of identifications. It is used to measure the accuracy of the system's container number recognition and reflects the core functional performance of the system. The collection efficiency can be defined as the number of container numbers effectively collected per unit time or the average time to complete a complete collection-identification process; is the false trigger rate, that is, the proportion of the system mistakenly triggering data collection or generating erroneous recognition results; the coefficients 0.6, 0.3 and -0.1 are the weight coefficients of recognition accuracy, collection efficiency and false trigger rate, respectively. They are set according to system requirements and actual application scenarios, reflecting the relative importance of each indicator in the system optimization goal. In the container number tally recognition system, since recognition accuracy is the primary goal, the recognition accuracy is given a higher weight; the collection efficiency is second; and false triggering will bring negative effects, so the false trigger rate is given a negative weight.

[0071] In the dynamic threshold triggering strategy, the two-layer DQN updates network parameters and adjusts the triggering strategy based on the reward value R obtained after each action (such as adjusting the collection interval and exposure time). For example, if the current strategy results in low recognition accuracy, the reward value will be reduced accordingly. The algorithm will adjust the strategy through learning to improve accuracy, thereby guiding the system to automatically optimize data collection timing and parameter settings in complex environments, improving collection efficiency and reducing false triggering rate while ensuring recognition accuracy.

[0072] (4) Training process: By interacting with the environment, continuously collecting data and calculating reward values, updating network parameters, and gradually optimizing the triggering strategy.

[0073] Implementation Example: Taking the morning rush hour at an automated terminal as an example, container truck traffic in the channel reaches twice its normal level, with generally faster speeds and closer spacing between vehicles. Three Advantech intelligent controllers interact in real time via Gigabit Ethernet ports to collaboratively monitor environmental conditions. The entropy of the current frame image is calculated to be 2.2 (low clarity), the variance of the depth map reaches 700 (high fluctuation), and the millimeter-wave radar point cloud density is 28 points / m³. Based on the dynamic threshold triggering strategy of the Deep Q Network, the controllers quickly shorten the acquisition interval of 16 industrial cameras from 200ms to 100ms via the 8-way GPIO interface, increase the exposure time of eight Intel RealSense D455 sensors from 6ms to 15ms, and simultaneously increase the data output frequency of four millimeter-wave radars and two lidars to 30Hz via the Gigabit Ethernet port. These adjustments are completed within 300 milliseconds, effectively reducing invalid data collection in scenarios with high-speed moving containers, significantly improving processing efficiency, and ensuring smooth operations during peak hours.

[0074] 2. Data fusion and preprocessing related models and algorithms:

[0075] 2.1 Cross-modal feature fusion network:

[0076] The cross-modal feature fusion network is based on the Transformer architecture and includes an eight-head multi-head self-attention mechanism. A dynamic routing mechanism adaptively assigns fusion weights by calculating the quantum entanglement similarity of features from different modalities (using the quantum state inner product formula). Quantum states represent high-dimensional mappings based on feature vectors. This method has been theoretically validated and experimentally tested to effectively capture the inherent connections between data from different modalities.

[0077] The edge computing unit adopts a heterogeneous architecture, including two NVIDIA RTX3060 Embedded GPUs for parallel processing of images and data; three Xilinx Zynq UltraScale+ MPSoC FPGAs for real-time data processing and control logic; four Cambricon MLU270 NPUs, focusing on deep learning inference; two Huawei Ascend 310 dedicated AI acceleration chips for processing specific algorithm tasks; among them, two NVIDIA GPUs and four Cambricon NPUs are installed in the high-performance computing module of the edge computing node, connected to the motherboard through the PCIe4.0 interface; three Xilinx FPGAs are distributed at the front end of data acquisition to achieve real-time pre-processing of data; two Huawei Ascend 310 chips are integrated on the algorithm acceleration board to work in conjunction with other devices. The details are as follows:

[0078] (1) Feature extraction layer: Use the ResNet-18 backbone network to extract features from RGB images, TOF depth images, millimeter wave point cloud data, and lidar point cloud data, and convert the original data into feature vectors.

[0079] (2) Multi-head self-attention mechanism: A multi-head self-attention mechanism with 8 heads is adopted, and the dimension of each head is 64. By calculating the attention weights between feature vectors of different modalities, the network can focus on multiple feature subspaces at the same time and capture the long-distance dependencies between data.

[0080] (3) Dynamic routing mechanism: Based on quantum entanglement similarity calculation, the quantum state inner product formula is used:

[0081] Calculate the similarity of different modal features, adaptively assign fusion weights, and achieve feature-level fusion.

[0082] Formula analysis: In quantum mechanics, quantum states can be represented by vectors, and the inner product is used to measure the similarity between two quantum states. This formula is calculated by and The square of the modulus of the inner product of is used to obtain the similarity measure between them. In the cross-modal feature fusion network, the features of different modalities are compared to quantum states. The formula is used to calculate the quantum entanglement similarity between the features to adaptively allocate the fusion weights.

[0083] and : Represents the "quantum state" vectors formed after encoding two different modal data (such as RGB image feature vectors and lidar point cloud feature vectors). These vectors contain the feature information of the corresponding modal data;

[0084] : represents the quantum state and The inner product of is calculated by multiplying the corresponding elements and then summing them (in vector representation). The result reflects the similarity and direction relationship between the two eigenvectors in the feature space.

[0085] : The final similarity value between the two modal features ranges from 0 to 1. The closer the value is to 1, the more similar the features of the two modalities are. In the feature fusion process, a higher weight should be given. When a modal feature (such as point cloud) is more important for target recognition in the current scenario (such as character occlusion scenario), its similarity increases and the corresponding weight automatically increases; conversely, the weight of the secondary modality (such as low-definition image) is automatically reduced to avoid invalid features interfering with the fusion results.

[0086] (4) Feature enhancement layer: The fused features are further enhanced through convolution operations and the final fused features are output.

[0087] Implementation example: At a container yard, the system collected data on a container with a highly reflective surface partially obscured by cargo. Two NVIDIA GPUs first used a ResNet-18 backbone network to extract feature vectors from RGB images captured by 16 Basler industrial cameras and point cloud data acquired by two Velodyne VLP-16 lidars. Subsequently, on four Cambricon NPUs, an eight-head Transformer architecture (with 512 hidden dimensions) parallelized the attention weights for features of different modalities. Analysis revealed that in the character localization task, the lidar point cloud features were highly significant for target recognition, reaching 0.75. Next, two Huawei Ascend 310s implemented a dynamic routing mechanism, further adjusting weights based on a quantum entanglement similarity formula. This increased the weight of the lidar point cloud features to 0.85, while reducing the weight of the RGB image features accordingly. Finally, three Xilinx FPGAs performed convolution operations to enhance the features, outputting fused features within 500 milliseconds. This feature accurately represented the location of the container number characters and some visible texture information, providing high-quality data support for subsequent character recognition and analysis.

[0088] 2.2. GAN-based denoising model:

[0089] The denoising model uses a U-Net structure generator and a PatchGAN structure discriminator. The adversarial training uses the LSGAN loss function. The generator and discriminator activation functions are LeakyReLU (slope 0.2), and the batch normalization layer momentum parameter is set to 0.9. These parameter configurations have demonstrated good denoising effects in a large number of data denoising experiments. The geometric distortion correction algorithm is based on a differentiable mapping model of differential homeomorphism. By minimizing the image reprojection error (threshold set to 1 pixel), optimizing the mapping parameters, correcting image distortion, ensuring the accuracy of data preprocessing, removing noise from multimodal data, improving data quality, and providing reliable input for subsequent processing. The details are as follows:

[0090] (1) Generator design: It adopts the U-Net structure, which contains 5 downsampling blocks and 5 upsampling blocks. The downsampling blocks extract image features through convolution and pooling operations, and the upsampling blocks restore image details through deconvolution and jump connections, and learn the mapping relationship between noisy images and clean images.

[0091] The generator and discriminator training is based on two NVIDIA RTX3060 Embedded GPUs, using a U-Net (5 downsampling + 5 upsampling blocks) structure generator and a PatchGAN structure discriminator. During the data processing process, the batch size is set to 32 and the learning rate is set to 2e-4.

[0092] Deployment method: Two NVIDIA GPUs are installed in the deep learning training module of the edge computing node and connected to the storage device via a high-speed data bus to ensure fast data transmission.

[0093] (2) Discriminator design: Using the PatchGAN structure, the input image is divided into multiple local patches, and each patch is judged to be from a real clean image or an image generated by the generator, thereby guiding the generator to generate more realistic images.

[0094] (3) Loss function design: The LSGAN (least squares generative adversarial network) loss function is adopted, including generator loss and discriminator loss. Through adversarial training, the generator can effectively remove data noise.

[0095] (4) Training process: Alternately train the generator and discriminator, and continuously adjust the network parameters until the generator can generate high-quality denoised images.

[0096] (5) Differentiable mapping model of diffeomorphism: Assume that the input image is , the output corrected image is , through the differentiable mapping function Establish pixel coordinate mapping relationship: ,in, are the distorted image coordinates, is the corrected coordinate, are learnable mapping parameters (such as translation, rotation, scaling, and nonlinear distortion coefficients).

[0097] (6) Reprojection error optimization: for each pixel in the image , calculate its coordinates after correction , and Reprojecting to the original image coordinate system yields , the reprojection error is: ; Minimize the reprojection error through the back propagation algorithm and update the mapping parameters , so that the error is below the 1 pixel threshold: When correcting visual images, the device posture information (such as pitch angle and yaw angle) in the IMU data is synchronously referenced, and the motion parameters of the inertial measurement are embedded in the mapping model to achieve real-time compensation of dynamic distortion. For example, when the IMU detects that the device vibration causes the camera to tilt 5°, the algorithm automatically adjusts the rotation component in the mapping parameter to offset the perspective distortion caused by the tilt.

[0098] (7) Distortion model initialization: obtain the camera’s internal parameters (focal length, principal point coordinates) and external parameters (rotation matrix, translation vector) through Zhang’s calibration method, and establish an initial distortion model (such as the Brown-Conrady model).

[0099] Differentiable mapping network construction, using convolutional neural network (CNN) architecture (such as U-Net encoding and decoding structure), input distorted image, output pixel-level offset ,Right now: ,in, is a learnable mapping network, is the network parameter.

[0100] End-to-end training uses paired distortion-corrected image datasets (e.g., synthetic distorted images + real-scene calibration images) and uses reprojection error as the loss function to ensure that the offset output by the network can accurately offset the distortion.

[0101] Real-time correction inference: For distorted images collected in real time, the trained network is used to predict the coordinate offset of each pixel and generate a corrected image. The processing delay is controlled within 100ms (to meet the real-time operation requirements of ports).

[0102] Implementation example: During a rainstorm, the RGB images of containers captured by 16 Basler industrial cameras were filled with noise due to dense raindrop adhesion and strong light reflection. This severely degraded image quality and extremely low clarity, making subsequent recognition extremely difficult. After these noisy images were input into the denoising model, two GPUs worked in parallel. The U-Net generator gradually extracted noise features and potentially useful information in the image through downsampling blocks. The upsampling blocks combined with jump connections gradually restored image details. The PatchGAN discriminator divided the image into 16×16 patches for true and false identification, and continuously fed back information to optimize the generator. After 50 epochs of training, the generator effectively removed raindrop noise, and the image entropy value significantly increased from 3.0 to 5.2. The character edges became clear and sharp, and the overall image quality was greatly improved, providing high-quality data for subsequent character recognition and improving recognition accuracy.

[0103] 3. Character recognition related models:

[0104] 3.1. Improved YOLOv7-tiny target detection model:

[0105] The model is deployed on four Huawei Ascend 310 dedicated AI acceleration chips in edge computing nodes, using a parallel computing architecture. The input size is set to 640×640. After 30% pruning, the number of parameters is reduced from 6.3M to 3.8M, greatly reducing the model size and significantly improving the inference speed. The four Huawei Ascend 310 chips are integrated on the dedicated acceleration board of the edge computing node and connected to the data processing module through a high-speed interface to achieve fast data transmission and efficient computing.

[0106] Basic model selection: YOLOv7-tiny is used as the basic model. This model has a lightweight structure and is suitable for fast inference on edge computing devices.

[0107] Progressive knowledge distillation: A pre-trained large-scale object detection model (such as YOLOv7) is used as the teacher model, and the temperature parameter is set to 4. The soft label knowledge of the teacher model is transferred to the student model (improved YOLOv7-tiny), guiding the student model to learn richer feature expressions.

[0108] Dynamic network pruning: Based on computing resources and detection accuracy requirements, the Taylor expansion is used to evaluate the importance of each convolution kernel. The model is pruned by 30% to remove redundant connections and parameters, reducing the number of model parameters by 40%. This improves detection speed while ensuring detection accuracy.

[0109] Introduction of the spatiotemporal attention mechanism: A spatiotemporal feature fusion module is added to the Feature Pyramid Network (FPN). Through the spatiotemporal feature pyramid structure (containing three scales, corresponding to 1 / 4, 1 / 8, and 1 / 16 resolution of the original image, respectively), combined with the spatiotemporal cross-attention module, the changing characteristics of the container number character area at different time steps are captured.

[0110] Analysis: Introducing time dimension analysis into the target detection process enhances the feature expression capability for fast-moving and small-sized character targets, and solves the problem of high missed detection rate of dynamic targets in traditional single-frame detection models.

[0111] Spatiotemporal feature pyramid structure: Based on the Feature Pyramid Network (FPN), it adds feature fusion in the time dimension to construct multi-scale and multi-time series feature expressions;

[0112] Spatiotemporal cross attention mechanism: By calculating the attention weights of spatial position and time step, the key feature areas of dynamic targets are highlighted.

[0113] 3.1.1, the structure of the spatiotemporal feature pyramid is as follows:

[0114] Multi-scale spatial features, using the FPN architecture of YOLOv7-tiny, output spatial feature maps at three scales (with resolutions of 1 / 4, 1 / 8, and 1 / 16 of the original image, respectively), corresponding to the detection of large, medium, and small-sized character targets.

[0115] Temporal dimension expansion: For each spatial scale feature map, the historical features of consecutive V frames (V=5) are introduced to form a spatiotemporal feature cube (spatial dimension H×W, temporal dimension T). The spatiotemporal feature cube is convolved through 3D convolution (kernel size 3×3×3, time step 1) to extract cross-frame motion features (such as the displacement trajectory of characters in consecutive frames).

[0116] 3.1.2, the spatiotemporal cross attention module is as follows:

[0117] Attention calculation logic, (1) Spatial attention: In a single-frame feature map, a spatial attention weight map is generated through convolution to highlight the areas where characters may exist (such as the fixed position of the container number); (2) Temporal attention: At the same spatial position in consecutive frames, feature similarity is calculated to generate temporal attention weights (for example, when a character at a certain position appears in three consecutive frames, the weight is increased);

[0118] Cross-fusion, multiply the spatial and temporal attention weights to obtain the spatiotemporal joint attention weight, the formula is: in, Represents element-by-element multiplication, and the final weight is used to adjust the fusion strength of spatiotemporal features. Represents the spatial attention weight matrix, with dimension H×W, which indicates the importance of each spatial position (i, j) in the image to target detection (the larger the value, the more important it is). Represents the temporal attention weight matrix with dimensions T×H×W, which indicates the stability of the features of the same spatial position (i, j) in T consecutive frames in the temporal dimension (the larger the value, the more stable it is).

[0119] Integration with LSTM: A two-layer LSTM network (hidden layer dimension 256) is connected to the top layer of the spatiotemporal feature pyramid (small-size feature map) to perform serialized modeling of temporal features and capture the movement patterns of characters over long time spans (such as the continuous displacement of a truck passing at high speed).

[0120] 3.1.3 Dynamic target trajectory tracking:

[0121] Through the fusion of spatiotemporal features, the module can track the motion trajectory of characters in five consecutive frames of images, calculate their displacement vectors in the image (with an accuracy of ±2 pixels), and combine them with IMU motion data (such as vehicle acceleration) to achieve precise positioning of high-speed moving targets.

[0122] Small target feature enhancement: For small character targets with a pixel area of ​​less than 50px², traditional single-frame detection is prone to missed detection due to insufficient features. The spatiotemporal feature fusion module accumulates weak features from multiple frames (such as blurred pixels at the same position in two consecutive frames) to form a detectable strong response area in the spatiotemporal feature cube, thereby improving the detection accuracy of small targets.

[0123] Improved anti-interference capabilities: A temporal attention mechanism filters out temporary noise (such as single-frame artifacts caused by sudden changes in lighting), retaining only stable features across multiple consecutive frames. This effectively prevents false detections in dynamic scenes. For example, when a truck passes through a brightly lit area, the module compares features across three consecutive frames to eliminate interference caused by overexposure in a single frame, maintaining stable detection results.

[0124] 3.1.4. Implementation process and parameter configuration:

[0125] Input data: 5 consecutive frames of images (resolution 640×640), spatial features are extracted through the backbone network (CSPDarknet); IMU data (acceleration, angular velocity) is used as temporal motion prior and input into the spatiotemporal cross attention module.

[0126] For feature processing, the spatiotemporal feature pyramid generates spatiotemporal feature maps of three scales (sizes are 160×160×5, 80×80×5, and 40×40×5). The spatiotemporal cross-attention module outputs the weighted feature maps and connects them to the detection head (classification layer + regression layer) of YOLOv7-tiny.

[0127] Graph neural network semantic association analysis: The character region detection results are constructed into a graph structure, where nodes represent character candidate boxes and edges represent semantic relationships between characters. A graph convolutional network (GCN) is used, which contains two graph convolution layers. Each layer has an output dimension of 128 and an activation function of ReLU. Node features are updated through graph convolution operations, semantic associations between characters are mined, candidate boxes that do not meet the rules are filtered out, and accurate character region positioning results are output.

[0128] Implementation example: A container truck traveling at a speed of 60 km / h enters the recognition area. A computing cluster consisting of four Huawei Ascend 310 chips receives five consecutive frames of image data. The improved YOLOv7-tiny model uses the rich features learned by progressive knowledge distillation to quickly identify character candidate areas within 80ms. The spatiotemporal attention mechanism captures the rapid displacement of characters in the image and accurately tracks the movement trajectory of characters through a three-level feature pyramid (1 / 4, 1 / 8, and 1 / 16 scales) and five frames of historical data. The graph neural network semantic association analysis module constructs the candidate box as a graph structure, analyzes it through a two-layer graph convolutional network (output dimension 128), and combines it with pre-built container number character semantic rules to filter out false detection areas that do not meet the rules. Finally, it accurately outputs the character area bounding box, effectively improving the detection accuracy in high-speed motion scenarios, successfully identifying the container number, and ensuring the accurate recording of logistics information.

[0129] During the pre-training phase of the improved YOLOv7-tiny object detection model, an expanded training set of 5,000 container number images with different lighting, angles, and sharpness was added to the COCO dataset. A multimodal pre-training strategy based on contrastive learning was adopted to construct an image-point cloud-text contrastive learning task. The contrastive loss was calculated using the InfoNCE loss function, and the temperature parameter was set to 0.07. This value has been verified by multiple sets of experiments to achieve the optimal effect in balancing feature discrimination and model stability. The positive-negative sample ratio was set to 1:3, which can effectively improve the model's ability to extract container number features and its generalization performance.

[0130] The multimodal pre-training strategy involves fusing multiple modal data (such as images, point clouds, and text) for joint learning in the early stages of model training. This allows the model to capture the correlation features between different modalities, thereby improving its generalization capabilities for complex scenarios. This strategy is primarily used in the pre-training phase of the improved YOLOv7-tiny object detection model. The specific implementation is as follows:

[0131] Data composition: 5,000 new RGB images of container numbers with different lighting (strong light / weak light), angles (tilted / straight ahead), and clarity (clear / blurred) were added; point cloud data from millimeter-wave radar and lidar were simultaneously collected to characterize the three-dimensional geometric features of the container; and the character semantic information of the container number (such as the ISO 6346 standard format rules) was extracted.

[0132] Through cross-modal comparative learning, the association between image pixels, point cloud spatial distribution and text semantics is established, so that the model can learn to understand the characteristics of container numbers from multiple dimensions.

[0133] Contrastive learning task design: forces the model to learn similar features of the same target in different modalities, while widening the feature distance between different targets. The details are as follows:

[0134] Positive sample pair: image-point cloud-text data combination of the same container number;

[0135] Negative sample pairs: any combination of modal data with different container numbers (positive and negative sample ratio is 1:3)

[0136] Loss function: InfoNCE loss function (noise contrast estimation loss) is used, and the calculation formula is:

[0137] ;

[0138] Where N is the number of query samples; K is the number of negative samples in the query samples, is the feature vector of the i-th query sample, is the feature vector of the i-th positive sample (point cloud / text features of the same sample), is the feature vector of the i-th negative sample (arbitrary modal features of different samples), =0.07 is the temperature parameter, which controls the concentration of characteristic distribution.

[0139] During the knowledge distillation stage, the teacher model is a complete YOLOv7 model with a temperature parameter set to 4. This temperature value is most effective in balancing knowledge transfer efficiency and model convergence speed, and soft labels are used to guide student model learning. Dynamic network pruning removes unimportant connections at a ratio of 30%, reducing the number of model parameters by 40%, significantly improving detection speed while maintaining high detection accuracy.

[0140] 3.2 Transformer-based cGAN character segmentation model:

[0141] The model runs on two Cambricon MLU270 NPUs (128TOPS computing power). The Transformer part uses a 4-head multi-head self-attention mechanism, and the adaptive receptive field can be dynamically adjusted within the range of 3×3-11×11 to adapt to different character forms. Among them, the two Cambricon MLU270 NPUs are installed in the AI ​​inference module of the edge computing node, which is closely connected with the data storage and transmission module to ensure efficient data processing and fast transmission. The details are as follows:

[0142] (1) Generator design: An adaptive receptive field module is introduced into the generator to dynamically adjust the receptive field size (ranging from 3×3 to 11×11) according to the stroke complexity and morphological characteristics of the characters. At the same time, a multi-head self-attention mechanism is used to capture the long-distance dependencies of characters and improve segmentation accuracy.

[0143] (2) Discriminator design: A PatchGAN structure similar to the GAN-based denoising model is used to judge the authenticity of the segmentation results.

[0144] (3) Post-processing of topological structure analysis: Calculate the topological invariants such as the Euler number and the number of holes in the connected area of ​​the characters (Euler number threshold is -1 to 1, and the number of holes threshold is 0 to 2), judge the integrity of the characters, repair broken characters, and remove noise points.

[0145] (4) Training process: Through adversarial training, the parameters of the generator and discriminator are continuously optimized so that the generator can accurately segment characters.

[0146] Implementation example: When an old container with rusty surface and severely adhered characters enters the recognition range, two Cambricon MLU270 NPUs work together. The generator of the cGAN model first detects the complex stroke structure of the adhered characters through the adaptive receptive field module and automatically adjusts the receptive field to 9×9 to better capture the character details. The multi-head self-attention mechanism fully plays its role, accurately identifying character boundaries and preliminarily segmenting the character areas. However, some characters are broken. The topological structure analysis module calculates the Euler number of the connected area of ​​the characters as -1 and the number of holes as 1, determining that the characters are broken. Morphological operations are performed to repair them and remove surrounding noise. After a series of processing, a clear and complete character segmentation image is finally output within 300 milliseconds, providing high-quality input for subsequent character recognition. Even for such severely damaged characters, they can be effectively segmented, laying the foundation for accurate recognition.

[0147] 3.3. Lightweight CNN-Transformer-Capsule Network Integration Model:

[0148] The lightweight CNN (MobileNetV3), Transformer, and capsule network are deployed in a collaborative processing architecture of two Cambricon MLU270 NPUs and two NVIDIA RTX3060 Embedded GPUs. The PPO algorithm is used for dynamic weight fusion, and the batch size is set to 64. Among them, the two Cambricon MLU270 NPUs are responsible for the calculation tasks of the Transformer and capsule network, and are installed in the AI ​​acceleration module of the edge computing node; the two NVIDIA RTX3060 Embedded GPUs handle MobileNetV3 related tasks and are deployed in the image processing module. The two realize data sharing and collaborative computing through a high-speed data bus. The details are as follows:

[0149] (1) Lightweight CNN design: Using the MobileNetV3 architecture, we utilize its inverted residual structure and SE attention mechanism to efficiently extract local features of characters and reduce the amount of model computation.

[0150] (2) Transformer design: It adopts a 6-layer encoder structure and uses a multi-head self-attention mechanism to capture the global semantic information of characters and understand the contextual relationship between characters.

[0151] (3) Capsule network design: 16 capsules are set, each with a dimension of 32, to extract the spatial hierarchical structure of characters and capture the relative position and posture information between character components.

[0152] (4) Bidirectional knowledge distillation: Adopting the adversarial knowledge transfer mechanism, ResNet50 is used as the teacher model to transfer knowledge to the integrated model. The student model optimizes the teacher model feature expression through adversarial training feedback. At the same time, the contrastive learning mechanism is introduced to enhance the model's ability to distinguish character features. The adversarial training process is as follows:

[0153] Feature extraction: The teacher model extracts features from the input image (such as the pooling layer output of ResNet50); the student model extracts features from the same image (Fused features of integrated model).

[0154] Discriminator training: input mixed features , the discriminator learns to distinguish the sources, and the loss function is the cross entropy: ,in, The discriminator outputs a probability (the closer it is to 1, the feature comes from the teacher model). E is the mathematical symbol for "taking the average of the samples according to the distribution" (the mathematical expectation represents the average operation of the random variable (sample distribution)). The loss of a single sample is generalized to the statistical law of the entire dataset, so that the loss function can reflect the performance of the model on the entire data.

[0155] Student model training forces student features to approach teacher features through adversarial loss, and the loss function is: That is, the student model tries to deceive the discriminator so that Close to 1.

[0156] The teacher model is indirectly optimized. During the adversarial process, the feature distribution of the teacher model will be adjusted by the discriminator to improve the feature distinction ability of difficult samples (such as ambiguous characters).

[0157] In bidirectional knowledge distillation, the specific process of knowledge transfer is as follows:

[0158] Feature alignment loss, combined with traditional knowledge distillation loss (soft label loss ) and adversarial loss , forming a comprehensive loss: ,in, (weights that can be determined experimentally), balancing semantic knowledge transfer and adversarial feature alignment.

[0159] Discriminator decision boundary, ideally, the discriminator cannot distinguish between teacher and student features, that is, , indicating that the student model has fully learned the teacher's feature distribution.

[0160] (5) Dynamic weight fusion strategy: Based on the reinforcement learning optimization framework, the PPO algorithm is used for training. The state space includes five dimensions, such as the prediction confidence of each sub-model and the computation time. The action space is the weight distribution of each model (0-1). The reward function is designed. , dynamically adjust the output weights of each model with the recognition accuracy and computational efficiency as the goal to achieve accurate classification and recognition of characters, where, Indicates the recognition accuracy of the model for the container number characters. The value range is between 0 and 1 and is used to measure the correctness of the model recognition result. Indicates the time it takes for the model to process a recognition task, and the unit can be seconds or other time units; The time consumption is converted into an indicator with the same direction as the accuracy, that is, the shorter the time consumption, the larger the value of this part.

[0161] Using this reward function, the reinforcement learning algorithm calculates a corresponding reward value during training based on the recognition accuracy and computational time obtained after each decision (weight allocation of each sub-model). This guides the model to learn the optimal weight allocation strategy for each sub-model. This allows the model to achieve fast and accurate container number character recognition while minimizing computational time while maintaining high recognition accuracy in different input scenarios.

[0162] Implementation example: To identify a container number, MobileNetV3 on two NVIDIA GPUs quickly extracts local texture features of the characters, such as subtle wear marks on the edges of the characters and unique printing textures; the Transformer on two Cambrian NPUs captures global semantic information through a 6-layer encoder to understand the sequential relationship and overall structure between characters; the capsule network deeply analyzes the spatial hierarchy of character components to determine the inclination angle and relative position of the characters. In the bidirectional knowledge distillation process, the pre-trained ResNet50 is used as the teacher model to guide the learning of the integrated model. During the recognition process, when encountering easily confused characters such as "9" and "6", the dynamic weight fusion strategy increases the Transformer weight from 0.3 to 0.5 in real time based on the predicted confidence and computational time of each sub-model, allowing it to play a greater role in the comprehensive judgment. Ultimately, the integrated model accurately identifies the container number within 120ms, effectively avoiding recognition errors caused by character confusion and ensuring the accurate entry of container information.

[0163] When constructing the character structure knowledge graph, the Neo4j graph database was used to manually annotate 2,000 groups of container number character samples, clarifying node attributes (including character type, size, spacing, etc.) and edge attributes (covering spatial relationships, semantic relationships, etc.), and pre-defined constraints such as character spacing of 1.2-1.5 times the character height and aspect ratio of 0.8-1.2. These parameters are based on industry standards for container number characters and statistics from a large number of actual samples, providing a precise screening basis for subsequent character area positioning.

[0164] In summary, in the hybrid model, the lightweight CNN adopts the MobileNetV3 architecture, the Transformer is a 6-layer encoder, the capsule network contains 16 capsules with a dimension of 32, and the bidirectional knowledge distillation adopts an adversarial knowledge transfer mechanism. The teacher model guides the student model to learn through soft labels, and the student model optimizes the teacher model through adversarial training feedback. The adversarial training discriminator adopts a multi-layer perceptron (2 hidden layers, 256 neurons per layer). The dynamic weight fusion strategy is based on the reinforcement learning optimization framework and adopts the PPO algorithm. The reward function is "recognition accuracy × 0.8 + calculation speed × 0.2". The weight setting of this reward function has been verified by multiple groups of comparative experiments and can effectively balance recognition accuracy and computational efficiency. The policy network and value network adopt a multi-layer perceptron (hidden layer size 256, training batch size 64), dynamically adjust the model weights, and improve the overall recognition performance.

[0165] 4. Intelligent verification and decision-making related models:

[0166] 4.1. Dynamic Bayesian Network Verification Decision Model:

[0167] The dynamic Bayesian network uses a reasoning mechanism that integrates evidence theory and Bayesian optimization. The basic probability distribution function of evidence theory is determined based on historical data statistics. The Bayesian optimization acquisition function is the expected improvement (EI) function. Format compliance verification is implemented through a probabilistic finite state automaton. The state transition probability is statistically determined based on historical data, and the threshold is set to 0.8. This threshold can effectively distinguish between compliant and non-compliant container number formats. Spatiotemporal consistency verification uses a spatiotemporal graph convolutional network (3 spatiotemporal convolutional layers, a time step of 5, and a spatial neighborhood size of 3×3). The targeted verification enhancement strategy uses a meta-verification framework, which includes 5 verification enhancement strategies (such as increasing the number of sampling times, switching recognition models, etc.). The probability of strategy selection is optimized through a genetic algorithm to ensure the accuracy and reliability of verification decisions. The recognition results are verified in multiple dimensions through probabilistic reasoning, the credibility of the results is evaluated, and reasonable decisions are made for abnormal situations. The details are as follows:

[0168] The model runs on a distributed computing cluster consisting of three Advantech UNO-3083 intelligent controllers (Intel Core i7-8550U processor, 16GB DDR4 memory). The dynamic Bayesian network contains 15 nodes (5 verification dimensions + 10 status nodes), and data sharing and collaborative computing are achieved through network communication. The three Advantech intelligent controllers are deployed in the channel monitoring center, equipment management room and edge computing station respectively, and connected by a gigabit local area network to form a distributed computing architecture to ensure efficient data transmission and collaborative processing.

[0169] Network structure construction: Based on the verification requirements of container number recognition, a dynamic Bayesian network consisting of multiple nodes and edges is constructed. The nodes represent different verification dimensions (such as format compliance, spatiotemporal consistency, etc.) and recognition results, and the edges represent the probabilistic dependency between nodes.

[0170] (1) Parameter initialization: By analyzing the historical container number recognition data, the prior probability distribution of each node is statistically analyzed, and the network parameters are initialized. For format compliance verification, a probabilistic finite state automaton containing 12 states is constructed. The state transition probability matrix is ​​obtained by training 100,000 historical data, and the threshold is set to 0.8.

[0171] (2) Fusion of evidence theory: Combine evidence theory with Bayesian networks, obtain the basic probability distribution function of evidence theory through historical data statistics, fuse evidence from different sources, and improve the accuracy of reasoning.

[0172] (3) Bayesian optimization: Using the expected improvement (EI) function, the network structure and parameters are dynamically updated according to real-time verification data to optimize the network's reasoning ability.

[0173] (4) Decision-making mechanism design: When the recognition result passes the verification, the confirmation information is output; when an exception occurs, the targeted enhancement strategy of the meta-verification framework is triggered.

[0174] Implementation example: The system identifies the container number "ABC1234X". A cluster consisting of three Advantech intelligent controllers works together to verify the recognition result. During format compliance verification, the probabilistic finite state automaton, based on the state transition probability matrix obtained by historical data training, determines that the number conforms to the ISO6346 standard format with a probability of 0.9. The spatiotemporal consistency verification module analyzes its motion trajectory for the past five frames and verifies the probability of 0.95. However, during multimodal data consistency verification, the Wasserstein distance of the RGB image, depth image, and point cloud data feature vectors is calculated and found to have large differences, resulting in a verification probability of only 0.4. The dynamic Bayesian network integrates the information of each node and the probability relationship for reasoning, and ultimately determines that the recognition result is abnormal, triggering the multi-view re-sampling strategy of the meta-verification framework to take measures to further confirm the accuracy of the recognition result.

[0175] 4.2. Meta-Authentication Framework Implementation:

[0176] The meta-verification framework is launched, and the policy library contains five policies: OCR post-processing (Python script calls Tesseract-OCR), multi-view data re-collection (controls the DJI Matrice 300RTK drone), historical data comparison (queries the MySQL database), sensor parameter recalibration, and manual intervention prompts.

[0177] A genetic algorithm (50 individuals, 100 generations) optimized strategy selection and calculated the fitness of each strategy based on historical data. Ultimately, the multi-perspective data recapture strategy was prioritized. The system then sent instructions to the DJI drone to recapture images from three angles: side and overhead.

[0178] After the supplementary data is processed by the edge computing device, it is re-identified and verified, and it is confirmed that the original result is wrong. The correct number is "ABC8765B".

[0179] The verified container number and related information are output to the tally system. At the same time, the blockchain evidence storage technology based on zero-knowledge proof is combined with the data processing method of homomorphic encryption to perform tamper-proof evidence storage. Among them, the zero-knowledge proof adopts zk-STARKs technology to generate 1024-bit proof, and the verification time is less than 100ms. The homomorphic encryption adopts the BGV homomorphic encryption scheme with a modulus q=2. 40 , the plaintext space is Z_2 10It supports ciphertext operations and ensures data privacy. The blockchain evidence storage adopts side chain technology. The side chain adopts the practical Byzantine fault tolerance (PBFT) consensus mechanism (the number of consensus nodes is 5, the Byzantine node tolerance number is 1). The evidence data is organized in a Merkle tree structure (depth 10). The side chain block size is 4MB, and the confirmation time is less than 2 seconds. The verified container number and related information will be output to the tallying system and stored as evidence to ensure data security and traceability.

[0180] 5. Implementation of system self-learning related algorithms:

[0181] The dynamic training dataset is constructed using an active learning strategy. Uncertainty is measured by predicting entropy (threshold 0.9) to select the most valuable error cases. The online learning strategy is based on the model-independent meta-learning (MAML) framework. The number of tasks in the meta-training phase is 1000, the meta-learning rate is 0.01, and the number of gradient updates in the meta-test phase is 2. The dynamic Bayesian network parameters are updated using the variational inference method, and the variational lower bound is constructed by minimizing the KL divergence. The trust region policy optimization (TRPO) algorithm is used to optimize the parameters of the reinforcement learning reward function. The trust region size is 0.01 and the optimization step size is 0.5. The system performance is continuously optimized to improve the recognition accuracy.

[0182] The implementation examples of the above models and algorithms are closely integrated with actual port operation scenarios, and detail the entire process from data collection to result verification, ensuring the integrity and operability of the overall technical solution, and providing a practical technical reference for container number tallying recognition in the field of smart logistics.

[0183] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for identifying container number tally, characterized in that: The identification method includes the following steps: S1. Deploy multimodal sensing devices and perform initial calibration, and configure reinforcement learning parameters. Pre-train an improved YOLOv7-tiny object detection model, a Transformer-based cGAN character segmentation model, and a hybrid recognition model integrating a lightweight CNN, Transformer, and capsule network to build a character structure knowledge graph and initialize the dynamic Bayesian network parameters. S2: Multi-modal sensing equipment with a self-calibration mechanism, combined with multi-sensor fusion positioning, synchronously collects multi-faceted and multi-modal data of the container; S3: Based on the improved YOLOv7-tiny object detection model, it integrates the spatiotemporal attention mechanism, character structure knowledge graph, and semantic association analysis based on graph neural network to detect character regions in multimodal data; S4. Utilize the Transformer-based cGAN character segmentation model, combined with morphological post-processing of topological structure analysis; perform character image segmentation and normalization through an adaptive normalization network; S5: A hybrid recognition model integrating lightweight CNN, Transformer, and capsule network is used, combining bidirectional knowledge distillation technology with a dynamic weight fusion strategy to perform character classification and output recognition results. S6. Construct a verification decision model based on a dynamic Bayesian network and combine it with a multi-dimensional verification strategy to evaluate the credibility of the recognition results; when a verification result is abnormal, initiate a targeted verification enhancement strategy.

2. The method for identifying container numbers according to claim 1, characterized in that: In S1, the multimodal sensing device includes a distributed stereo vision array, millimeter-wave radar, and lidar, and a timestamp synchronization mechanism is used to calibrate the time of the multimodal sensing device; Among them, the distributed stereo vision array is calibrated with internal and external parameters using Zhang's calibration method; Millimeter-wave radar and lidar use NDT algorithm for point cloud registration.

3. The method for identifying container numbers according to claim 2, characterized in that: In S2, multimodal data is acquired through a distributed stereo vision array with a self-calibration mechanism, combined with multi-sensor fusion positioning of millimeter-wave radar and lidar. RGB images, time-of-flight depth images, millimeter-wave point cloud data, and lidar point cloud data are simultaneously collected from multiple sides of the container. Dynamic threshold triggering and adaptive exposure control strategies based on a deep Q network are used to obtain multimodal data including the container number area. Among them, the dynamic threshold triggering strategy adopts a two-layer Q network architecture to achieve multi-objective optimization, including the target network and the real network. The target network parameters are updated every L steps, and the real network uses the Adam optimizer, L>100; The adaptive exposure control strategy dynamically adjusts the exposure time by calculating the image entropy value.

4. The method for identifying container numbers according to claim 1, wherein: In S2, the self-calibration mechanism uses a checkerboard calibration plate and IMU deployed in the transport channel to perform real-time spatiotemporal calibration of the distributed stereo vision array, millimeter-wave radar, and lidar based on the Kalman filter algorithm. The Kalman filter's state transition matrix is ​​determined by the IMU's motion model, and the observation matrix is ​​determined by the checkerboard corner detection results. The distributed stereo vision array adopts a non-uniform redundancy design and dynamically adjusts the sensor layout density according to the recognition requirements of key areas of the container. The sensor density in key areas is 2 to 3 times that of non-key areas.

5. The method for identifying container numbers according to claim 1, characterized in that: Before character region detection, a cross-modal feature fusion network based on the Transformer architecture is used, combined with a dynamic routing mechanism, to fuse multi-source data at the feature level. A noise reduction model based on a generative adversarial network and a geometric distortion correction algorithm based on deep learning are then used to pre-process the fused data. The cross-modal feature fusion network includes a multi-head self-attention mechanism, and the dynamic routing mechanism adaptively allocates fusion weights by calculating the quantum entanglement similarity of different modal features. The similarity calculation uses the quantum state inner product formula. The denoising model consists of a generator and a discriminator. The generator adopts a U-Net structure, which contains 5 downsampling blocks and 5 upsampling blocks. The discriminator adopts a PatchGAN structure, and the loss function of adversarial training is LSGAN. The geometric distortion correction algorithm adopts a differentiable mapping model based on diffeomorphism and optimizes the mapping parameters by minimizing the image reprojection error. The threshold of the reprojection error is set to 1 pixel. The generator and discriminator of the denoising model both use the LeakyReLU activation function.

6. The method for identifying container numbers according to claim 1, characterized in that: In S3, the improved YOLOv7-tiny model uses a collaborative optimization architecture of progressive knowledge distillation and dynamic network pruning. The spatiotemporal attention mechanism captures the changing characteristics of character regions at different time steps through a spatiotemporal feature pyramid structure. The feature pyramid contains three scales, corresponding to 1 / 4, 1 / 8, and 1 / 16 resolutions of the original image. The semantic association analysis of graph neural network uses graph convolutional network, which includes 2 graph convolution layers. The output dimension of each layer is 128 and the activation function is ReLU. The dynamic update of the character structure knowledge graph is achieved through incremental learning. After each recognition result is updated, the confidence score of the knowledge graph is recalculated, and nodes and edges with a confidence score lower than 0.5 are deleted.

7. The method for identifying container numbers according to claim 1, characterized in that: In S4, the segmentation and normalization of character images are achieved through an adaptive normalization network based on a differentiable deformation module. The differentiable deformation module adopts a hybrid deformation model that combines bilinear interpolation and thin plate spline transformation. The adaptive normalization network adopts an attention-guided spatial transformer network. The attention weight map is generated by the Softmax function, and the parameters of the spatial transformer network are learned through back propagation.

8. The method for identifying container numbers according to claim 1, characterized in that: In S5, bidirectional knowledge distillation uses an adversarial knowledge transfer mechanism. The teacher model guides the student model through soft labels. The student model optimizes the feature expression of the teacher model through adversarial training feedback. The discriminator of adversarial training uses a multi-layer perceptron with two hidden layers and 256 neurons in each layer. The dynamic weight fusion strategy adopts a reinforcement learning optimization framework to dynamically adjust the model weights with recognition accuracy and computational efficiency as optimization goals. The reward function of reinforcement learning is recognition accuracy × 0.8 + computational speed × 0.

2.

9. The method for identifying container numbers according to claim 1, characterized in that: In S6, the dynamic Bayesian network adopts an inference mechanism that integrates evidence theory and Bayesian optimization. The basic probability distribution function of evidence theory is obtained through historical data statistics, and the acquisition function of Bayesian optimization is the expected improvement function. The targeted verification enhancement strategy adopts a meta-verification framework to automatically select the optimal verification enhancement solution based on the verification anomaly type. The meta-verification framework includes five verification enhancement strategies and optimizes the strategy selection probability through a genetic algorithm. Format compliance verification is implemented through a probabilistic finite state automaton, and the state transition probability of the automaton is obtained through historical data statistics; The spatiotemporal consistency verification is constructed using a spatiotemporal graph convolutional network. The graph convolutional network contains 3 spatiotemporal convolutional layers, the time step is set to 5, and the spatial neighborhood size is 3×3.

10. The method for identifying container numbers according to claim 1, characterized in that: The method further comprises the following steps: S7. Output the verified container number and related information to the tally system. Simultaneously, the blockchain evidence storage technology based on zero-knowledge proof and combined with homomorphic encryption data processing methods are used to store the information in an unalterable manner. Among them, zero-knowledge proof uses zk-STARKs technology; Homomorphic encryption uses the BGV homomorphic encryption scheme, which supports addition and multiplication operations on ciphertext; Blockchain evidence storage uses sidechain technology for high-frequency transaction processing; The side chain of blockchain evidence storage adopts a practical Byzantine fault-tolerant consensus mechanism.

Images