Logistics loading and unloading monitoring system and method based on deep neural network visual recognition

Abstract

The present application relates to the technical field of logistics loading and unloading, in particular to a logistics loading and unloading monitoring system and method based on deep neural network visual recognition. It comprises the following steps: collecting panoramic and local operation video streams, and simultaneously obtaining weight data and environmental state data related to loading and unloading operations; through a lightweight deep neural network model deployed on an edge computing node, structured recognition feature data is generated to determine the current state of the loading and unloading operation process; the lightweight deep neural network model is called to extract visual emptying features inside the carriage in real time, and consistency verification is performed in combination with the change rate characteristics of the weight data to generate a loading and unloading completion signal and trigger process state switching. By fusing the carriage floor pixel ratio features extracted by visual segmentation and the change rate characteristics of the weight sensor, the accuracy of the emptying state determination is ensured under various extreme light and complex cargo forms, and the reliability of the supervision data is improved.

Description

Technical Field

[0001] This invention relates to the field of logistics loading and unloading technology, and more specifically, to a logistics loading and unloading monitoring system and method based on deep neural network visual recognition. Background Technology

[0002] With the continuous development of international trade, the safety and efficiency of cargo transportation have received increasing attention. Traditional methods of loading and unloading supervision suffer from inefficiency and inaccurate data recording, and can no longer meet the needs of modern international trade. Therefore, the construction of an intelligent loading and unloading supervision system is particularly important. It can improve the transparency and safety of cargo transportation, reduce cargo losses and disputes, and promote the smooth progress of international trade. To improve customs clearance efficiency and meet regulatory requirements, it is necessary to analyze the loading and unloading process of logistics goods, promptly detect abnormal behaviors, and achieve real-time monitoring, recording, and handling of the cargo loading and unloading process.

[0003] However, the loading and unloading process of logistics goods involves many key elements such as license plates, vehicle locks, vehicle compartments, workers, and goods. These elements are diverse in type and form, making identification complex. For example, the shape, size, and color of goods vary greatly. Most existing intelligent monitoring systems rely on pure vision solutions for status determination, such as judging whether the cargo compartment is empty by recognizing images inside the compartment. However, in dim lighting, strong backlighting, or when the color of the goods is similar to the floor of the compartment, pure vision solutions are prone to misjudgment (e.g., mistaking leftover goods for shadows on the floor). Therefore, this paper proposes a logistics loading and unloading monitoring system and method based on deep neural network visual recognition. Summary of the Invention

[0004] The purpose of this invention is to provide a logistics loading and unloading monitoring system and method based on deep neural network visual recognition, so as to solve the problem mentioned in the background art that pure visual solutions are prone to misjudgment in the case of dark, strong backlight or cargo color similar to the truck bed.

[0005] To achieve the above objectives, the present invention aims to provide a logistics loading and unloading monitoring method based on deep neural network visual recognition, comprising the following steps: S1. Collect panoramic and local operation video streams, and simultaneously acquire weight data and environmental status data related to loading and unloading operations to form a time-synchronized multimodal raw dataset. S2. The multimodal raw dataset is transmitted to the edge computing node. A lightweight deep neural network model deployed on the edge computing node is used to detect and track vehicle identity, carriage structure status, personnel and cargo targets in the video stream in real time, and generate structured recognition feature data. S3. Based on the structured identification feature data, determine the current state of the loading and unloading operation process, and update the loading and unloading operation process state according to the preset process state transition rules. The loading and unloading operation status includes the operation status of vehicle arrival, loading and unloading in progress, loading and unloading completed and awaiting confirmation, and vehicle departure. S4. When the operation process is in the loading and unloading operation, the lightweight deep neural network model is called to extract the visual emptying features inside the car in real time, and the consistency is checked by combining the change rate features of weight data. When both the visual features and weight features meet the preset emptying threshold conditions, a loading and unloading completion signal is generated and the process state is switched. S5. During each loading and unloading operation process, conduct parallel monitoring of abnormal personnel intrusion and illegal operation behaviors, and generate compliance judgment results.

[0006] As a further improvement to this technical solution, the multimodal raw dataset includes: Loading and unloading time The corresponding set of multi-view video frames; Real-time weight data collected by a weight sensor; A set of environmental parameters for loading and unloading operations, including at least temperature and humidity data for the storage area; Location information parameters of the operating equipment, including truck coordinates.

[0007] As a further improvement to this technical solution, in S2, the lightweight deep neural network model includes at least a multi-scale convolutional feature extraction module, an attention-weighted feature enhancement module, and a multi-target classification and regression output head, and the multi-target classification and regression output head includes at least a license plate recognition sub-module, a vehicle lock recognition sub-module, an unloading determination sub-module, and a violation behavior recognition sub-module.

[0008] As a further improvement to this technical solution, the specific steps involved in generating structured recognition feature data through the lightweight deep neural network model are as follows: The collected loading and unloading operation video stream is input into the multi-scale convolutional feature extraction module. The convolutional kernels with different receptive fields encode the features of vehicles, carriages, workers and cargo targets in the image, and generate a multi-scale basic feature map to represent the spatial structure information of the loading and unloading operation scene. The multi-scale basic feature map is input into the attention-weighted feature enhancement module, and the feature responses of at least one type of region, including the interior area of ​​the carriage, the vehicle lock area, and the personnel activity area, are weighted to obtain a weighted feature map. Based on the weighted feature map, multiple targets in the loading and unloading operation scenario are identified and located in parallel through a multi-target classification and regression output head, and vehicle identity information, vehicle lock status information, empty compartment status information and abnormal operation behavior information are output respectively. Based on the results of multi-objective classification and regression, the corresponding functional sub-modules are called to make specific judgments. The identification and judgment results of the above functional sub-modules are uniformly encoded to generate structured identification feature data that includes at least vehicle identification, unloading status indicators and violation behavior indicators.

[0009] As a further improvement to this technical solution, the specific process of calling the corresponding functional sub-module for special determination is as follows: The license plate recognition submodule extracts vehicle license plate characters based on optical character recognition technology, associates them with vehicle declaration information, and generates a vehicle identification identifier. ; The vehicle lock identification submodule identifies the physical state of the vehicle lock based on a binary classification judgment model, and determines the physical state of the truck door lock; The unloading determination submodule extracts the pixel percentage features of the empty areas inside the carriage and combines them with synchronously collected weight sensor data to generate multi-dimensional unloading status indicators. ; The violation identification submodule is used to detect and mark abnormal behaviors such as cross-selling and unauthorized personnel entering the work area, and to generate a violation identification status vector. .

[0010] As a further improvement to this technical solution, the preset process state transition rules in S3 are specifically as follows: When vehicle identification When a match is successful and no violations are detected, the process status enters the loading / unloading in progress state. When the vehicle is in position and a loading / unloading operation start signal is detected, the process status switches to the loading / unloading operation in progress state. When multi-dimensional unloading status indicators When the preset unloading threshold is met, the loading and unloading process is determined to have transitioned from the loading and unloading operation to the loading and unloading completed and pending confirmation state. Once the loading and unloading is completed and the pending confirmation status is confirmed to be normal, the process status will switch to the vehicle departure status. In any state, if the system detects a violation, it will determine the current process state as an abnormal event state.

[0011] As a further improvement to this technical solution, the specific steps involved in generating the loading / unloading completion signal in step S4 are as follows: Obtain the bounding box of the cargo compartment of the current working vehicle based on structured recognition feature data. And capture images of the interior of the carriage. ; Foreground segmentation is performed on the image inside the carriage to obtain a mask for the area without cargo. And calculate the pixel proportion feature of the area without goods. ; Synchronously collect real-time weight values ​​output by the carriage weight sensor and the unloaded calibrated weight Normalization yields weight features ; Based on the pixel proportion feature of the unloaded area By introducing weight characteristics, a multi-dimensional unloading status index is constructed. ; And when multi-dimensional unloading status indicators At that time, it was determined that the carriage was empty; When the car is determined to be empty, a loading / unloading completion signal is generated to trigger the loading / unloading operation process status to switch from the loading / unloading operation in progress state to the loading / unloading completion pending confirmation state.

[0012] As a further improvement to this technical solution, the specific steps involved in generating the compliance determination result in step S5 are as follows: At the current loading and unloading operation time Next, receive the violation identification status vector output by the violation identification submodule. and the violation behavior identification status vector As input for the violation status at the current operation time; Based on the aforementioned violation identification state vector, a compliance determination logic for loading and unloading operations is constructed. When any condition is met, the current loading and unloading operation is determined to be non-compliant. When the work process is in the loading / unloading operation stage and is determined to be non-compliant, an abnormal work event identifier is generated and an alarm is triggered. When the work process status is in the loading and unloading operation completed state and the work compliance judgment result is compliant, the work process status is allowed to switch to the vehicle departure state. Based on vehicle identification, work process status sequence, loading and unloading completion signal and work compliance judgment result, a work event chain is constructed from vehicle entry, loading and unloading operations to departure, and the work event chain and corresponding violation and compliance monitoring data are synchronously uploaded to the cloud supervision platform.

[0013] On the other hand, the present invention provides a logistics loading and unloading monitoring system based on deep neural network visual recognition, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the logistics loading and unloading monitoring method based on deep neural network visual recognition described above, characterized in that it includes: A multimodal acquisition module is used to control a fixed camera device, a weight sensor, an environmental sensor, and a mobile acquisition device to acquire time-synchronized multimodal raw datasets. An edge computing module, based on a lightweight deep neural network model, is used to receive the multimodal raw dataset and generate structured recognition feature data. The process status control module is used to receive structured recognition feature data and to determine and update the loading and unloading operation process status in real time based on preset process status transition rules. The closed-loop monitoring module is used to combine the loading / unloading completion signal and the status vector of the violation behavior identifier to generate closed-loop monitoring data for the entire process and synchronize it to the cloud monitoring platform.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This logistics loading and unloading monitoring system and method based on deep neural network visual recognition innovatively introduces a gradient-preserving constraint loss function and a scale compensation factor into a lightweight deep neural network. This addresses the problem of truncation of gradient values ​​for tiny targets (with extremely low pixel proportions) such as vehicle locks during mixed-precision training due to excessively small gradient values, forcing the model to preserve the gradient response of these tiny features. Simultaneously, combined with an attention-weighted feature enhancement module, weights are allocated to dark areas and key target areas inside the vehicle during the feature extraction stage, significantly suppressing environmental background noise. Compared to traditional general detection algorithms, this invention significantly improves the detection rate and recognition accuracy of tiny targets in complex lighting and dynamic occlusion scenarios.

[0015] 2. In this logistics loading and unloading monitoring system and method based on deep neural network visual recognition, a visual and weight multimodal consistency verification is constructed; the pixel ratio feature of the truck bed extracted by visual segmentation is fused with the change rate feature of the weight sensor, and the weight data is used to logically verify the visual subjective data, which ensures the accuracy of the unloading status determination under various extreme lighting and complex cargo shapes, and improves the reliability of the supervision data. Attached Figure Description

[0016] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: Please refer to Figure 1 As shown, this embodiment provides a logistics loading and unloading monitoring method based on deep neural network visual recognition, including the following steps: S1. Within the logistics loading and unloading area, panoramic and local operation video streams are collected using fixed camera equipment, and mobile operation acquisition equipment is used to dynamically supplement the fixed field of view blind spots. Simultaneously, weight data and environmental status data related to the loading and unloading operation are acquired to form a time-synchronized multimodal raw dataset. The multimodal raw dataset includes: Loading and unloading time The corresponding multi-view video frame set (including top-down and side-view images acquired by a fixed camera array, and blind-filling images acquired by a panoramic camera of a collaborative robot); furthermore, in harsh lighting environments such as strong light and heavy fog, a 16-line lidar (point cloud density ≥100 points / ㎡) is preferred to acquire the three-dimensional structure of the carriage to assist in the unloading determination. Real-time weight data is collected at a frequency of 1Hz by weight sensors deployed in the weighing area of ​​the unloading platform, with an accuracy of ±0.5kg. The set of environmental parameters for loading and unloading operations includes at least the temperature and humidity data of the storage area collected by temperature and humidity sensors; The location information parameters of the operating equipment include the truck coordinates obtained through the GPS positioning module (where, when the truck coordinates fall within the preset operating geographical coordinate range and the coordinate values ​​remain stationary for a continuous time (e.g., 10 seconds), the system can automatically switch the process state to the vehicle in position state (Q1), thereby triggering subsequent camera OCR recognition) and the robot's real-time pose data (through the robot pose data, the standard alignment of the visual sensor (camera) coordinate system and the robot's mechanical coordinate system can be achieved (hand-eye calibration); based on the robot's pose data, the key area of ​​cargo handling in the next frame of video is predicted in real time, guiding the deep neural network to prioritize the processing of this local area, thereby reducing the computational pressure on edge computing nodes and improving the recognition frame rate).

[0019] In this embodiment, the fixed camera equipment is deployed in key areas such as the unloading platform, the entrance and exit of the inspection area, and the area around the loading and unloading platform, covering the truck parking area, the cargo stacking area, and the sorting line entrance. Each area is equipped with 2-3 cameras at different angles (horizontal, downward and side views). The top-view camera deployed on the platform and the side-view camera deployed on the side of the platform are used to collect data on the status of goods inside the carriage and the vehicle's parking position and lock status, respectively. The mobile data collection device is a multi-functional collaborative robot equipped with a 2-DOF robotic arm (load 5kg) and a 360° surround-view panoramic camera (2K resolution). The multi-functional collaborative robot adopts a wheeled mobile chassis, with a maximum speed of 1.5m / s and a battery life of ≥8h.

[0020] S2. The multimodal raw dataset is transmitted to the edge computing node. A lightweight deep neural network model deployed on the edge computing node is used to detect and track vehicle identity, carriage structure status, personnel and cargo targets in the video stream in real time, and generate structured recognition feature data. The lightweight deep neural network model includes at least a multi-scale convolutional feature extraction module, an attention-weighted feature enhancement module, and a multi-target classification and regression output head. The multi-target classification and regression output head includes at least a license plate recognition sub-module, a vehicle lock recognition sub-module, an unloading determination sub-module, and a violation behavior recognition sub-module.

[0021] In this embodiment, the specific steps involved in generating structured recognition feature data through the lightweight deep neural network model are as follows: The collected loading and unloading operation video stream is input into the multi-scale convolutional feature extraction module. The convolutional kernels with different receptive fields encode the features of vehicles, carriages, workers and cargo targets in the image, and generate a multi-scale basic feature map to represent the spatial structure information of the loading and unloading operation scene. In this embodiment, in order to balance real-time performance ( ) and small goals ( To improve pixel detection accuracy, the improved CSPDarknet-53 is preferred as the backbone network, and multi-scale fusion is performed in conjunction with FPN (Feature Pyramid Network). During loading and unloading operations The collection of multi-view loading and unloading operation video frames obtained is as follows:

[0022] In the formula, Indicates the first RGB video frames from various perspectives Indicates the height of the video frame. Indicates the width of the video frame; This indicates the number of viewpoints participating in the recognition simultaneously, preferably This includes at least the top-down and side-view perspectives captured by fixed cameras, as well as the blind-filling perspectives captured by collaborative robots. After uniformly scaling and normalizing the pixels of the video frames from all perspectives (Resize to 640*640*3, normalize to...), the resulting images are then processed to achieve the desired resolution. The input is fed into a multi-scale convolutional feature extraction module (which integrates an improved CSPDarknet-53 backbone network, based on the original CSPDarknet-53 architecture, containing 53 layers of convolutional networks and several CSP (Cross Stage Partial) modules. By adding extra convolutional layers and residual connections to shallow convolutional branches, it improves the feature response capability for small targets such as car locks and people's heads; at the same time, a feature pyramid network is introduced to perform hierarchical feature encoding of image targets through convolutional kernels with different receptive fields, realizing effective feature representation of small, medium and large targets). Specifically, by introducing a Feature Pyramid Network (FPN), which uses convolutional kernels with different receptive fields to perform hierarchical feature encoding on image targets, at least three layers of multi-scale basic feature maps with different spatial resolutions are extracted from the input video frames. These are used to characterize small-scale, medium-scale, and large-scale targets, respectively. Specifically, small-scale feature maps The dimensions are 80*80*256, and the receptive field is 8*8 pixels, which is used to encode fine-grained texture features of vehicle locks and workers; Mesoscale feature map The dimensions are 40*40*512, and the receptive field is 16*16 pixels, used to encode the geometric features of cargo units; Large-scale feature maps The dimensions are 20*20*1024, and the receptive field is 32*32 pixels, used to encode the spatial structural information of the vehicle body and the outline of the carriage. The multi-scale basic feature map The input is fed into the attention-weighted feature enhancement module, which assigns weights to the feature responses of at least one type of region, including the interior region of the carriage, the vehicle lock region, and the personnel activity region, to enhance the feature representation ability of the key target region and suppress the interference features of the background region, thereby obtaining a weighted feature map. In this embodiment, a spatial attention weight map is constructed for key areas in the loading and unloading operation scenario by introducing an attention-weighted feature enhancement module: ; In the formula, For attention mapping function, by Attention mapping functions composed of convolution and non-linear activation (such as ReLU or SiLU) are used to learn the importance weights of each position in the feature map; The Sigmoid activation function is used to normalize the attention mapping values ​​to the [0,1] interval; This represents the original feature map to be weighted (for) or ); To and A spatial attention weight map of the same size, where each element represents the importance weight of the corresponding pixel position; Weighting the original features: ; in, Represents element-wise multiplication; the weighted feature map Used for subsequent target identification, specifically... This represents a small-scale weighted feature map. This represents a mesoscale weighted feature map. Represents a large-scale weighted feature map; This represents the weighted feature map, which is used for subsequent multi-target classification and regression output. Based on the weighted feature map Through parallel multi-target classification and regression output head, multiple targets in the loading and unloading operation scenario are identified and located in parallel, and vehicle identity information, vehicle lock status information, empty compartment status information and abnormal operation behavior information are output respectively. In this embodiment, for each scale feature map, in order to eliminate feature conflicts between classification and regression tasks, independent classification and regression branches are constructed to decouple features: ; ; ; In the formula, Let be the target category probability vector, representing the predicted probability of each category at each spatial location; The number of categories includes vehicles, locks, people, etc. This represents the weights of the classification convolution kernel, used to map feature maps to class predictions; This represents the normalization of the category vector for each spatial location, outputting a probability distribution; Indicates the categorical branch bias term; This represents the regression output feature map, used to predict the parameters of the target bounding box; This represents the regression convolution kernel weight matrix; Indicates the regression bias term; Here are the parameters of the target bounding box, where, The coordinates of the bounding box center are Define the width and height of the bounding box; This is the grid offset, used to map the relative offset to the corresponding spatial location in the feature map; The prior bounding box size is used for scale decoding of the bounding box. This represents the Sigmoid function, used to constrain the predicted offset between [0,1]. An exponential mapping representing the width of the target bounding box predicted by the regression branch, used to scale the network output. Convert to the actual width ratio to ensure that the bounding box width is a positive value; The exponential mapping representing the height of the target bounding box predicted by the regression branch is used to scale the network output. Convert to the actual height ratio to ensure that the bounding box height is a positive value; This represents the horizontal offset (relative position) of the bounding box predicted by the regression branch within the feature map grid cell, mapped to the grid offset using a Sigmoid algorithm. Adding them together gives the horizontal position of the bounding box center in the image coordinates; This represents the vertical offset (relative position) of the bounding box predicted by the regression branch within the feature map grid cell, mapped to the grid offset using a Sigmoid algorithm. Adding them together gives the vertical position of the bounding box center in the image coordinates; The logarithmic scaling adjustment of the bounding box width for regression branch predictions, relative to the prior box width. Adjust the width of the predicted bounding box, after exponential mapping. The actual width is then obtained; The logarithmic scaling adjustment of the bounding box height predicted by the regression branch, relative to the prior box height. Adjust the predicted bounding box height, after exponential mapping. The actual height was then obtained; In this embodiment, the vehicle identity information, vehicle lock status information, cargo compartment emptying status information and abnormal operation behavior information are output based on the multi-target classification and regression output head, and all types of information are uniformly encoded to form structured identification feature data, which is used to drive the loading and unloading operation process status control and compliance judgment. Furthermore, to adapt to edge-side mixed-precision inference environments, this embodiment preferably employs a CANN architecture based on the Ascend AI processor, which performs small-scale feature map training during the training phase. Gradient preservation constraints are introduced to prevent the vanishing of key feature gradients during mixed precision training (FP16), thereby improving the stability of small target recognition. Specifically, the gradient cutoff threshold for small-scale target feature layers is locked during backpropagation. ,in, To preset the gradient threshold, the preferred method is... ), Representing small-scale feature maps The original gradient during the backpropagation phase is used to calculate the gradient update parameters; Indicates the result after gradient truncation Feature map gradient, which is constrained to prevent the gradient from being too small and affecting training stability; The gradient cutoff function is defined as restricting the input value to a certain range. Within the range, ensure the gradient value remains stable. This represents a preset gradient threshold, used to define the upper and lower limits of gradient cutoff, and the preferred range. This ensures that key small target features do not experience gradient explosion during mixed-precision training; Specifically, to address the issue of tiny targets (occupying less than 1% of the image area) such as vehicle locks and people's heads being truncated due to excessively small gradient values ​​during mixed-precision training in logistics scenarios, this embodiment introduces gradient preservation constraints during the training phase:

[0023] In the formula, The actual area of ​​the frame A small set of pixels; This represents the scale compensation factor. ,in, The width of the input image. The height of the input image. Indicates the first The width of each predicted or ground truth bounding box. Indicates the first The height of the predicted or ground truth bounding box for each target; This represents the small target loss function, used to enhance the supervision signal and improve the recognition stability for targets with an area smaller than 32×32 pixels during training; , which is a weighting coefficient used to adjust the contribution ratio of the small target loss to the overall loss; Indicates the first The prediction confidence of a small target is used to measure the probability that the network can recognize that target. This represents the Focal Loss adjustment parameter, used to suppress the contribution of easily classified samples to the loss and enhance the focus on small, hard-to-classify targets; The core calculation term representing cross-entropy is used to measure the difference between the predicted distribution and the true distribution; For extremely small targets, the scale compensation factor approaches 2, which is used to amplify their gradient response value during backpropagation, making it higher than the minimum cutoff threshold for FP16 accuracy. This ensures that the feature updates of tiny targets do not stagnate; Based on the results of multi-objective classification and regression, the corresponding functional sub-modules are called to make specific judgments. The identification and judgment results of the above functional sub-modules are uniformly encoded to generate structured identification feature data that includes at least vehicle identification, unloading status indicators, and violation behavior indicators. ;in, For vehicle identification, This is an indicator of the unloading status. This serves as a vector for identifying violations; the structured identification feature data is used as the input basis for subsequent driving of loading and unloading operation process status control and compliance determination.

[0024] Furthermore, the corresponding functional sub-modules are invoked for specific judgment, and the specific process involved is as follows: The license plate recognition submodule extracts vehicle license plate characters based on optical character recognition (OCR) technology, associates them with vehicle declaration information, and generates a vehicle identification identifier. Furthermore, for scenarios where vision is obstructed, RFID technology can be used to replace optical character recognition (OCR) technology. Passive tags can be attached to vehicle locks, and card readers can be deployed on the platform to identify the status of vehicle locks through radio frequency signals. The vehicle lock recognition submodule identifies the physical state of the vehicle lock based on a binary classification judgment model, and determines whether the physical state of the truck door lock is locked or unlocked. In this embodiment, the bounding box of the vehicle lock target is output based on the multi-target recognition results. Capture the image of the car lock area: ; In the formula, This represents the bounding box of the vehicle lock target, output by the multi-target recognition module, used to extract the image of the vehicle lock region; Indicates the current time Input video frames (RGB images); This represents the image of the car lock area, which is the car lock image cropped from the video frame; Represents image cropping operations, based on bounding boxes. Crop the image; The vehicle lock area is input to the vehicle lock recognition submodule, and the physical state of the vehicle lock is identified by a binary classification model (using a binary classification neural network with a convolutional + fully connected structure). ; in, , This represents the probability of a locked state. Indicates the probability of being in an unlocked state; when the condition is met... If the lock status is positive, the vehicle lock is considered locked; otherwise, it is considered unlocked. The threshold value is used to determine the probability that the car lock is locked. The preferred setting is 0.6 to 0.8; In the formula, The mapping function for the vehicle lock recognition submodule uses a binary classification neural network composed of convolutional and fully connected structures; express The activation function maps the network output to a normalized probability vector. In this embodiment, the vehicle identification is based on the results of multi-target recognition and vehicle lock recognition. With car lock status A unified coding system forms structured identification feature data, which is used to drive the status control and compliance determination of loading and unloading operations.

[0025] The unloading determination submodule extracts the pixel percentage features of the empty areas inside the carriage and combines them with synchronously collected weight sensor data to generate multi-dimensional unloading status indicators. ; The violation identification submodule is used to detect and mark abnormal behaviors such as cross-selling and unauthorized personnel entering the work area, and to generate a violation identification status vector. : Specifically, the continuous temporal location trajectory of cargo targets generated based on a multi-target tracking algorithm (based on a multi-target tracking algorithm, for continuous time intervals) and The collected cargo targets are associated to construct a continuous temporal location trajectory for each cargo target: In the formula, Indicates the first The cargo target at time Spatial position vector, This indicates the current time frame captured in the loading / unloading operation video. Indicates the number of historical frames used to construct continuous temporal location trajectories (preferred). The spatial position vector is preferably represented as the center coordinates of the target bounding box. In the formula, Indicates the lateral position of the cargo target in the image coordinate system; (This indicates the longitudinal position of the cargo target in the image coordinate system). When the movement trajectory of any cargo target at an adjacent time point crosses the preset left and right work area boundary lines, And when the vehicle identifier currently associated with the cargo target does not match the identifier of the vehicle currently operating on the platform (assuming preset left and right work area boundaries). For planar segmentation boundaries, when the positions of the same cargo target at adjacent time points satisfy... ), marked as parallel import behavior Otherwise adopt This indicates that no cross-selling behavior was detected. Represents position vector The position sign (which can be positive or negative) of the work area relative to the boundary is used to determine the boundary crossing situation; When any personnel target is detected entering the preset restricted work area If the identification result of the target person does not match the system's preset list of authorized personnel, the system determines that the target person has engaged in unauthorized entry and generates a corresponding violation entry behavior identifier. Otherwise, a corresponding normal entry behavior identifier will be generated. ; At the same loading and unloading time The following steps are performed to identify the parallel importing behavior according to the preset one-to-one correspondence between violation event types and vector dimensions. Signs indicating unauthorized entry Perform unified encoding to generate a violation behavior identifier state vector. , represents a unified coding violation identification status vector, corresponding to two types of violation events (parallel import, unauthorized entry); Furthermore, if the vehicle identification currently associated with the cargo does not match the identification of the platform operation vehicle, it is determined to be an illegal cross-delivery.

[0026] S3. Based on the structured identification feature data, determine the current state of the loading and unloading operation process, and update the loading and unloading operation process state according to the preset process state transition rules. The loading and unloading operation status includes the operation status of vehicle arrival, loading and unloading in progress, loading and unloading completed and awaiting confirmation, and vehicle departure. In this embodiment, the vehicle's positioning status ( When OCR recognizes the vehicle identification identifier And the vehicle has been stationary for more than 10 seconds, the status is (That is, collect the real-time coordinates of the truck and spatially match them with the preset platform geofence; if the truck coordinates are inside the geofence and meet the stationary time threshold, then the current process state is determined to enter the vehicle arrival state.) )); Status during loading and unloading operations ( When the vehicle lock is detected to be unlocked or a sudden change in weight is detected, the state switches to... ; And in the state during loading and unloading operations ( The violation identification submodule works in parallel: Conduct anti-counterfeiting assessment: track the trajectory of goods. If the trajectory crosses the virtual boundary Furthermore, the cargo ownership does not match, so a violation vector is marked. ; Intrusion prevention assessment: If unauthorized personnel appear in the restricted work area. ,mark ; Loading and unloading completed, pending confirmation status ( When the above unloading determination conditions are met ( When loading / unloading is complete, a loading / unloading completion signal is generated, and the status is switched to... ; Vehicle departure status ( The vehicle target disappeared and If a vehicle is lost, the process is marked as finished and all monitoring data is packaged and uploaded to the cloud. Furthermore, GPS positioning can be used to determine the vehicle's departure status. The process ends when the vehicle leaves the platform area by ≥5 meters, which is suitable for open-air operation scenarios.

[0027] In this embodiment, the preset process state transition rules are as follows: When vehicle identification When a match is successful and no violation is detected (i.e., when the violation identification state vector is in the correct state), (All components are 0), the process status enters the loading / unloading in progress state. ; When the vehicle is in position (current status is vehicle in position) When a loading / unloading operation start signal is detected, the process status remains unchanged or switches to the loading / unloading operation in progress state. ; When multi-dimensional unloading status indicators When the preset emptying threshold is met, the loading and unloading process is determined to have transitioned from the loading and unloading operation state. Transferred to the status of loading / unloading completed and awaiting confirmation. ; Loading / unloading completed and awaiting confirmation. After confirming that there are no abnormalities, the process status is switched to vehicle departure status. ; Specifically, in this embodiment, a lightweight deep neural network model is used to perform real-time detection of the vehicle compartment during loading and unloading operations, and outputs: ; In the formula, The bounding box of the vehicle compartment is used to locate the target's position in the image; The confidence score represents the reliability of the model's detection of the vehicle compartment. The logic for dynamic confidence level switching is as follows: Set the detection confidence threshold (Preferred value: 0.3~0.5) is used to determine whether the carriage can be effectively detected. ; in, This indicates that the carriage inspection was reliable. This indicates that the carriage inspection failed or the confidence level was too low.

[0028] At the same time, a continuous time counter is introduced. :

[0029] in, This is an indicator function that takes the value 1 when the condition is true, and 0 otherwise. This indicates the cumulative time (in seconds) during which no carriage was detected consecutively. Indicates the current video frame capture time; Indicates the start time of the timing (used to initialize the consecutive loss count when the carriage detection fails for the first time); This represents the sampling time interval, measured in seconds (s), indicating the time difference between consecutive frames. When the duration of the carriage cannot be detected, If the vehicle has left the scene, it is determined that the vehicle has left the scene.

[0030] The loading and unloading operation status has been changed from "loading and unloading completed, pending confirmation" to "loading and unloading completed, pending confirmation". Automatically switch to vehicle departure ( The system is in a state that triggers the process to end and generates a vehicle departure signal.

[0031] In any state, when the system detects a violation (when any violation identifier component is 1, such as cross-selling or unauthorized personnel entering the work area), the current process state is determined to be an abnormal event state identifier, which is used to trigger abnormal intervention or alarm. The abnormal event state identifier is not one of the loading and unloading operation process states, and does not change the current loading and unloading operation process state set.

[0032] S4. When the work process is in the loading / unloading phase, a lightweight deep neural network model is invoked to extract the visual emptying features inside the car body in real time. This is combined with the rate of change of weight data for consistency verification. When both the visual and weight features meet the preset emptying threshold (the preset emptying threshold is an empirical threshold determined based on the statistical characteristics of multi-dimensional emptying status indicators under empty conditions (statistical characteristics include at least the sample mean and sample dispersion, obtained through statistical analysis of multiple sets of continuously collected emptying status indicators under empty conditions), preferably, the preset emptying threshold is set to 0.75~0.95), a loading / unloading completion signal is generated, triggering a process state switch. The specific steps involved are as follows: Obtain the bounding box of the cargo compartment of the current working vehicle based on structured recognition feature data. And capture images of the interior of the carriage. In the formula, This represents the bounding box of the current working vehicle's cargo compartment, used to locate the compartment's position in the image; Indicates the first Raw video frames captured in real time; images of the interior of the train carriage. Used to analyze the unloading status; This indicates an image cropping operation, which extracts the target region based on the bounding box. Foreground segmentation is performed on the image inside the carriage, and a mask for the area without cargo is calculated using a foreground segmentation algorithm. And calculate the pixel proportion feature of the area without goods. , represents the proportion of the empty area inside the carriage relative to the total number of pixels in the carriage; where, This indicates the number of pixels in the hollow area of ​​the mask; This indicates the total number of pixels in the carriage image; in, This represents the foreground segmentation function, used to divide the interior image of the carriage into cargo areas and non-cargo areas. Specifically, in this embodiment, the foreground segmentation function... A lightweight deep convolutional neural network (such as a foreground segmentation network based on the U-Net structure) is used as input for the interior image of the train carriage. Output a binary mask with the same size as the image. The pixel value of 1 represents an area without cargo, and 0 represents an area covered by cargo. The foreground segmentation network based on the U-Net structure is trained through supervised learning. The training data includes images of the interior of the carriage with cargo and empty carriages. Synchronously collect real-time weight values ​​output by the carriage weight sensor and the unloaded calibrated weight Normalization yields weight features : ; Based on the pixel proportion feature of the unloaded area A multi-dimensional unloading status index is constructed by introducing normalized weight features. ,in, Preferred , ; This represents the visual feature weighting coefficient, a parameter used to adjust the importance of the pixel proportion feature of the unloaded area in the final unloading determination index; This represents the weight feature weighting coefficient, a parameter used to adjust the importance of normalized weight features in the final unloading determination index; And when multi-dimensional unloading status indicators At that time, it was determined that the carriage was empty; In the formula, Indicates the first The real-time weight value output by the car's weight sensor at all times; This indicates the empty rated weight, which is the reference weight of the carriage when it is not loaded with goods; This represents the minimum constraint operator, used to impose an upper limit on the normalization result; To preset the unloading threshold, in this embodiment, a preset unloading threshold is defined. The results were obtained through experimental calibration, specifically by analyzing the multi-dimensional unloading status indicators collected under empty vehicle conditions. mean with standard deviation Statistical analysis was performed to determine the threshold. Among them, the preferred Preferably, The value is set between 0.75 and 0.95 to take into account the accuracy of unloading determination for different car body sizes and cargo types; When the car is determined to be empty, a loading / unloading completion signal is generated to trigger the loading / unloading operation process status to switch from the loading / unloading operation in progress state to the loading / unloading completion pending confirmation state.

[0033] Preferably, in this embodiment, to address environmental interference such as slight dust, water mist, or drastic fluctuations in light (where humidity data is used to pre-correct visual confidence; when humidity exceeds a preset threshold (e.g., 90%), the system determines that fog interference may exist), a dynamic weight adjustment mechanism based on environmental confidence is further introduced into the multi-dimensional unloading judgment logic to ensure that the system's judgment accuracy is maintained without relying on additional hardware (e.g., RFID). The specific steps are as follows: Calculate the current time Features of the percentage of pixels in the non-cargo area within the first N frames (preferably 20 frames) The sliding standard deviation was calculated, and a fluctuation threshold was preset (by sampling normal loading and unloading scenarios (no dust), the pixel ratio of the area without cargo was calculated). The moving standard deviation distribution under standard conditions (calculate the mean of its moving standard deviation sequence). with standard deviation And calculate the preset fluctuation threshold. ,in This is the confidence level adjustment coefficient. The preferred value range is 2 to 3, and the upper limit of the 95% confidence interval of this distribution is used as the preset fluctuation threshold. The preset fluctuation threshold is preferably 0.03 to 0.10. When the sliding standard deviation is greater than the preset fluctuation threshold, the visual segmentation result is determined to have abnormal fluctuations (i.e., the pixel proportion feature of the area without goods is judged). Within a short time window, during normal loading and unloading, the change in pixel percentage should be monotonic or smooth; dust interference, on the other hand, manifests as high-frequency oscillations, leading to an increase in the sliding standard deviation. When the dust is heavy, the pixel ratio of the visually segmented area without goods becomes a feature. When fluctuations occur (reduce the weight of visual features, select the best option) Due to weight feature weight Taking a dominant position, enabling multi-dimensional unloading status indicators It can still maintain consistency with the actual physical weight change, effectively avoiding incorrect process state switching triggered by visual misjudgment (such as identifying dust clumps as residual goods). (Specifically, when the sliding standard deviation shows extremely drastic visual fluctuations,) When the weight is reduced to the lower limit of 0.2, the weight feature weights are... Automatically increases to the upper limit of 0.8, and the system enters weight-driven and vision-assisted mode.

[0034] S5. During each loading and unloading operation process, conduct parallel monitoring of abnormal personnel intrusion and illegal operation behavior, generate compliance judgment results, and combine the operation process status and loading and unloading completion signal to complete the closed-loop monitoring of the entire process of vehicle entry to exit, and synchronize the monitoring data to the cloud supervision platform.

[0035] Specifically, the steps involved in generating a compliance determination result are as follows: At the current loading and unloading operation time Next, receive the violation identification status vector output by the violation identification submodule. and the violation identification status vector As input for the violation status at the current operation time; Based on the aforementioned violation identification state vector, a compliance determination logic for loading and unloading operations is constructed. When any condition is met, the current loading and unloading operation is determined to be non-compliant. The specific loading and unloading operation compliance determination logic is as follows: When the work process is in the loading / unloading operation stage and is determined to be non-compliant, an abnormal operation event identifier is generated, and an alarm or regulatory prompt is triggered. When the work process status is in the loading and unloading operation completed state and the work compliance judgment result is compliant, the work process status is allowed to switch to the vehicle departure state.

[0036] Based on vehicle identification, work process status sequence, loading and unloading completion signal and work compliance judgment result, a work event chain is constructed from vehicle entry, loading and unloading operations to departure, and the work event chain and corresponding violation and compliance monitoring data are synchronously uploaded to the cloud supervision platform.

[0037] In this embodiment, the work process state sequence is obtained by the continuous determination and update process of the current state of the loading and unloading operation in step S3; specifically, during the loading and unloading operation, at continuous moments... Below, based on structured recognition feature data Determine the current status of the loading and unloading operation process (within the time window). Within the system, the output workflow status at each time point is recorded as follows: ,in, Indicates at time The loading and unloading operation status obtained from the determination (including at least vehicle identification) Carriage boundary frame 1. Unloading status indicators of the carriage Status of violations Detection confidence The process state is updated according to preset process state transition rules, thereby forming a sequence of process states arranged in chronological order. This is a sequence of work process states, arranged in chronological order, representing consecutive moments. The status of the internally determined work process.

[0038] Example 2: This example provides a logistics loading and unloading monitoring system based on deep neural network visual recognition, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the logistics loading and unloading monitoring method based on deep neural network visual recognition described above. The system is characterized by including: A multimodal acquisition module is used to control a fixed camera device, a weight sensor, an environmental sensor, and a mobile acquisition device to acquire time-synchronized multimodal raw datasets. An edge computing module, based on a lightweight deep neural network model, is used to receive the multimodal raw dataset and generate structured recognition feature data. The process status control module is used to receive structured recognition feature data and to determine and update the loading and unloading operation process status in real time based on preset process status transition rules. The closed-loop monitoring module is used to combine the loading / unloading completion signal and the status vector of the violation behavior identifier to generate closed-loop monitoring data for the entire process and synchronize it to the cloud monitoring platform.

[0039] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A logistics loading and unloading monitoring method based on deep neural network visual recognition, characterized in that, Includes the following steps: S1. Collect panoramic and local operation video streams, and simultaneously acquire weight data and environmental status data related to loading and unloading operations to form a time-synchronized multimodal raw dataset. S2. The multimodal raw dataset is transmitted to the edge computing node. A lightweight deep neural network model deployed on the edge computing node is used to detect and track vehicle identity, carriage structure status, personnel and cargo targets in the video stream in real time, and generate structured recognition feature data. S3. Based on the structured identification feature data, determine the current state of the loading and unloading operation process, and update the loading and unloading operation process state according to the preset process state transition rules. The loading and unloading operation status includes the operation status of vehicle arrival, loading and unloading in progress, loading and unloading completed and awaiting confirmation, and vehicle departure. S4. When the operation process is in the loading and unloading operation, the lightweight deep neural network model is called to extract the visual emptying features inside the car in real time, and the consistency is checked by combining the change rate features of weight data. When both the visual features and weight features meet the preset emptying threshold conditions, a loading and unloading completion signal is generated and the process state is switched. S5. During each loading and unloading operation process, conduct parallel monitoring of abnormal personnel intrusion and illegal operation behaviors, and generate compliance judgment results.

2. The logistics loading and unloading monitoring method based on deep neural network visual recognition according to claim 1, characterized in that, The original multimodal dataset includes: Loading and unloading operations The corresponding set of multi-view video frames; Real-time weight data collected by a weight sensor; A set of environmental parameters for loading and unloading operations, including at least temperature and humidity data for the storage area; Location information parameters of the operating equipment, including truck coordinates.

3. The logistics loading and unloading monitoring method based on deep neural network visual recognition according to claim 1, characterized in that, In S2, the lightweight deep neural network model includes at least a multi-scale convolutional feature extraction module, an attention-weighted feature enhancement module, and a multi-target classification and regression output head. The multi-target classification and regression output head includes at least a license plate recognition sub-module, a vehicle lock recognition sub-module, an unloading determination sub-module, and a violation behavior recognition sub-module.

4. The logistics loading and unloading monitoring method based on deep neural network visual recognition according to claim 3, characterized in that, The specific steps involved in generating structured recognition feature data using the lightweight deep neural network model are as follows: The collected loading and unloading operation video stream is input into the multi-scale convolutional feature extraction module. The convolutional kernels with different receptive fields encode the features of vehicles, carriages, workers and cargo targets in the image, and generate a multi-scale basic feature map to represent the spatial structure information of the loading and unloading operation scene. The multi-scale basic feature map is input into the attention-weighted feature enhancement module, and the feature responses of at least one type of region, including the interior area of ​​the carriage, the vehicle lock area, and the personnel activity area, are weighted to obtain a weighted feature map. Based on the weighted feature map, multiple targets in the loading and unloading operation scenario are identified and located in parallel through a multi-target classification and regression output head, and vehicle identity information, vehicle lock status information, empty compartment status information and abnormal operation behavior information are output respectively. Based on the results of multi-objective classification and regression, the corresponding functional sub-modules are called to make specific judgments. The identification and judgment results of the above functional sub-modules are uniformly encoded to generate structured identification feature data that includes at least vehicle identification, unloading status indicators and violation behavior indicators.

5. The logistics loading and unloading monitoring method based on deep neural network visual recognition according to claim 4, characterized in that, The specific process involved in calling the corresponding functional sub-module for specific determination is as follows: The license plate recognition submodule extracts vehicle license plate characters based on optical character recognition technology, associates them with vehicle declaration information, and generates a vehicle identification identifier. ; The vehicle lock identification submodule identifies the physical state of the vehicle lock based on a binary classification judgment model, and determines the physical state of the truck door lock; The unloading determination submodule extracts the pixel percentage features of the empty areas inside the carriage and combines them with synchronously collected weight sensor data to generate multi-dimensional unloading status indicators. ; The violation identification submodule is used to detect and mark abnormal behaviors such as cross-selling and unauthorized personnel entering the work area, and to generate a violation identification status vector. .

6. The logistics loading and unloading monitoring method based on deep neural network visual recognition according to claim 5, characterized in that, In S3, the preset process state transition rules are specifically as follows: When vehicle identification When a match is successful and no violations are detected, the process status enters the loading / unloading in progress state. When the vehicle is in position and a loading / unloading operation start signal is detected, the process status switches to the loading / unloading operation in progress state. When multi-dimensional unloading status indicators When the preset unloading threshold is met, the loading and unloading process is determined to have transitioned from the loading and unloading operation to the loading and unloading completed and pending confirmation state. Once the loading and unloading is completed and the pending confirmation status is confirmed to be without any abnormalities, the process status will switch to the vehicle departure status. In any state, if the system detects a violation, it will determine the current process state as an abnormal event state.

7. The logistics loading and unloading monitoring method based on deep neural network visual recognition according to claim 4, characterized in that, In step S4, the specific steps involved in generating the loading / unloading completion signal are as follows: Obtain the bounding box of the cargo compartment of the current working vehicle based on structured recognition feature data. And capture images of the interior of the carriage. ; Foreground segmentation is performed on the image inside the carriage to obtain a mask for the area without cargo. And calculate the pixel proportion feature of the area without goods. ; Synchronously collect real-time weight values ​​output by the carriage weight sensor and the unloaded calibrated weight Normalization yields weight features ; Based on the pixel proportion feature of the unloaded area By introducing weight features Constructing multi-dimensional unloading status indicators ; And when multi-dimensional unloading status indicators At that time, it was determined that the carriage was empty; When the car is determined to be empty, a loading / unloading completion signal is generated to trigger the loading / unloading operation process status to switch from the loading / unloading operation in progress state to the loading / unloading completion pending confirmation state.

8. The logistics loading and unloading monitoring method based on deep neural network visual recognition according to claim 5, characterized in that, In step S5, generating the compliance determination result involves the following specific steps: At the current loading and unloading operation time Next, receive the violation identification status vector output by the violation identification submodule. and the violation identification status vector As input for the violation status at the current operation time; Based on the aforementioned violation identification state vector, a compliance determination logic for loading and unloading operations is constructed. When any condition is met, the current loading and unloading operation is determined to be non-compliant. When the work process is in the loading / unloading operation stage and is determined to be non-compliant, an abnormal work event identifier is generated and an alarm is triggered. When the work process status is in the loading and unloading operation completed state and the work compliance judgment result is compliant, the work process status is allowed to switch to the vehicle departure state. Based on vehicle identification, work process status sequence, loading and unloading completion signal and work compliance judgment result, a work event chain is constructed from vehicle entry, loading and unloading operations to departure, and the work event chain and corresponding violation and compliance monitoring data are synchronously uploaded to the cloud supervision platform.

9. A logistics loading and unloading monitoring system based on deep neural network visual recognition, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the logistics loading and unloading monitoring method based on deep neural network visual recognition as described in any one of claims 1-8, characterized in that, include: A multimodal acquisition module is used to control a fixed camera device, a weight sensor, an environmental sensor, and a mobile acquisition device to acquire time-synchronized multimodal raw datasets. An edge computing module, based on a lightweight deep neural network model, is used to receive the multimodal raw dataset and generate structured recognition feature data. The process status control module is used to receive structured recognition feature data and to determine and update the loading and unloading operation process status in real time based on preset process status transition rules. The closed-loop monitoring module is used to combine the loading / unloading completion signal and the status vector of the violation behavior identifier to generate closed-loop monitoring data for the entire process and synchronize it to the cloud monitoring platform.

Images