Embedded port edge intelligent identification methods, identification terminals, equipment and control systems

Abstract

This application provides an embedded port edge intelligent recognition method, recognition terminal, equipment, and control system, applied to the field of license plate and container basic information recognition technology in port operations. By receiving port operation video streams captured by industrial cameras, the video streams are converted into image frames and key frames are filtered out. A high-precision multi-target detection is performed using an improved YOLOv8 target detection model, and corresponding recognition models are called for different categories of labels. This achieves efficient and accurate recognition of information such as license plates, container numbers, dangerous goods labels, lead seals, and damage of trucks and containers in port scenarios. The overall recognition latency is reduced to less than 200 ms, and the accuracy rate is increased to over 98%. This significantly reduces the number of servers, lowers deployment and maintenance costs, and provides efficient technical support for port automation management and intelligent scheduling.

Description

Technical Field

[0001] This application relates to the technical field of license plate and container basic information recognition in port operations, specifically to an embedded port edge intelligent recognition method, recognition terminal, equipment and control system. Background Technology

[0002] Currently, the hardware and software solutions for port OCR recognition mainly adopt a centralized architecture: multiple high-definition network cameras (deployed at multiple angles including front, rear, left, right and center), ground loop coils or video analysis triggers are installed at the recognition site to assist in image acquisition. Industrial-grade servers or algorithm industrial control computers are deployed in the computer room. OCR recognition algorithms are configured on these servers or algorithm industrial control computers, and the OCR recognition algorithms are run using the server CPU or the computing card / graphics card of the algorithm industrial control computer.

[0003] However, since these recognition models run on PCs, their recognition speed is relatively slow, typically taking more than one second per recognition, which cannot meet the fast-paced requirements of continuous port passage or crane operations. Moreover, traditional solutions based on image capture require high-quality images, which can easily lead to high rates of missed and false detections in actual operation. In addition, industrial-grade servers or algorithm-based industrial control computers are expensive, bulky, and consume a lot of power. If port operations require the deployment of multiple such devices simultaneously, the overall cost will increase significantly.

[0004] Therefore, a new port OCR recognition solution is needed. Summary of the Invention

[0005] In view of this, embodiments of this specification provide an embedded port edge intelligent identification method, identification terminal, device and control system, which achieves millisecond-level identification of port vehicles and containers through improved target detection model and license plate recognition model.

[0006] The embodiments in this specification provide the following technical solutions:

[0007] In a first aspect, embodiments of this specification provide an embedded port edge recognition method, executed on an embedded edge device deployed at a port site, comprising:

[0008] Receive port operation video streams captured by industrial cameras, the video streams including: trucks and containers;

[0009] The video stream is converted into image frames and stored in the shared memory area;

[0010] Based on the adaptive frame sampling algorithm, key frames are selected from the consecutive image frames;

[0011] The keyframes are input into the improved target detection model to obtain the target detection regions annotated with rotated polygon boxes, and the category labels corresponding to each target detection region;

[0012] The improved target detection model is constructed in the following way:

[0013] The backbone network is based on the YOLOv8 architecture;

[0014] Add the WIoU v3 loss function to dynamically focus on hard samples;

[0015] Integrate a scale-compensated feature pyramid in the Neck layer to enhance multi-scale feature fusion;

[0016] The output layer supports rotating polygon frames to adapt to oblique angle shots of containers;

[0017] The category labels include: license plate, container number, dangerous goods label, lead seal, and damage;

[0018] The target detection area is identified. If the category label is a license plate, container number, or dangerous goods label, the license plate recognition model is called to perform character or pattern recognition on the target detection area. If the category label is a lead seal or damage, the improved target detection model is used to classify and identify the target detection area.

[0019] The identification results and corresponding location coordinates are output to the port control system.

[0020] Furthermore, the step of filtering the image frames based on the adaptive frame sampling algorithm includes:

[0021] Set the first image frame of the video stream as the initial keyframe and extract feature points;

[0022] Perform the following checks on each subsequent image frame:

[0023] The optical flow method tracks the position of feature points from the previous keyframe in the current frame frame by frame. If the proportion of successfully tracked feature points is lower than the preset tracking rate threshold, the processing of the current frame is terminated.

[0024] Based on the successfully tracked feature points, the mean optical flow vector and covariance matrix of all successfully tracked feature points are obtained. If the trace of the covariance matrix is ​​greater than a preset motion stability threshold, the current frame processing is terminated.

[0025] The proportion of optical flow anomalies is counted. If the proportion of anomalies is greater than the preset occlusion tolerance threshold, the processing of the current frame is terminated.

[0026] Calculate the Laplacian operator variance of the current frame. If the Laplacian operator variance is less than the preset sharpness threshold, terminate the processing of the current frame.

[0027] If the current frame passes all the criteria, set the current frame as the new keyframe and update the feature points.

[0028] Furthermore, the embedded port edge recognition method also includes:

[0029] When a container truck enters the inspection area, an industrial camera is triggered by a photoelectric sensor to capture images of the truck and / or container, thus obtaining keyframes.

[0030] The keyframes are stored in the shared memory area after being hardware decoded.

[0031] Furthermore, the target detection model and the license plate recognition model are deployed on the NPU processor.

[0032] Furthermore, the target detection model is converted into an INT8-quantized RKNN format model;

[0033] And / or, convert the license plate recognition model into an INT8 quantized RKNN format model.

[0034] Furthermore, before identifying the target detection region, the target detection region is parallelized using an OpenMP-accelerated C++ post-processor, including performing geometric correction and non-maximum suppression filtering on the rotated polygonal box.

[0035] Secondly, embodiments of this specification also provide an embedded edge recognition terminal for implementing the embedded port edge recognition method as described above, the embedded edge recognition terminal comprising:

[0036] The communication interface unit includes a gigabit Ethernet interface and an RS485 interface; wherein, the gigabit Ethernet interface is used to receive port operation video streams captured by industrial cameras, the video streams including trucks and containers; the RS485 interface is used to transmit the identification results and corresponding location coordinates to the port control system.

[0037] The processing unit adopts an industrial-grade domestic processor chip, which integrates a hardware decoding module, an NPU processor, and a CPU processor.

[0038] The hardware decoding module is used to convert the video stream into image frames and store them in a shared memory area; the NPU processor is used to deploy an improved target detection model and license plate recognition model; and the CPU processor is used to run the Linux operating system.

[0039] A customized circuit board, which is matched with the processing unit, includes a multi-layer PCB board; the customized circuit board is used to carry the communication interface unit, the processing unit, and the memory of the storage unit.

[0040] Furthermore, the embedded edge recognition terminal also includes: a dual gigabit PHY chip and an isolation circuit;

[0041] The dual gigabit PHY chip is connected to the processing unit to form a redundant second gigabit Ethernet interface to support automatic switching between fiber optic and copper cables.

[0042] The isolation circuit is used to block electromagnetic interference in the port.

[0043] Thirdly, embodiments of this specification also provide a port operation control system, the port operation control system comprising:

[0044] At least one industrial camera, a gigabit network switch, an embedded edge recognition terminal as described above, and a PLC controller;

[0045] The industrial camera is deployed at level crossings, rail-mounted gantry cranes, and / or bridge cranes, and is used to acquire real-time video streams of trucks and containers;

[0046] The gigabit network switch is used to connect the industrial camera to the embedded edge recognition terminal via gigabit Ethernet;

[0047] The embedded edge recognition terminal is used to recognize the video stream and output the recognition result and the corresponding position coordinates;

[0048] The PLC controller receives the recognition result and corresponding position coordinates from the embedded edge recognition terminal, and controls the barrier gate or hoisting equipment to perform corresponding operations based on the recognition result.

[0049] Fourthly, embodiments of this specification also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the embedded port edge recognition method as described above.

[0050] Compared with the prior art, the beneficial effects that at least one technical solution adopted in the embodiments of this specification can achieve include at least:

[0051] By receiving port operation video streams captured by industrial cameras, the video streams are converted into image frames and key frames are selected. A high-precision multi-target detection model is then used with an improved YOLOv8 target detection model. The corresponding recognition model is called for different categories of labels, which enables efficient and accurate identification of information such as license plates, container numbers, dangerous goods labels, lead seals, and damage of trucks and containers in port scenes. The overall recognition latency is reduced to less than 200 ms, and the accuracy rate is increased to over 98%. This significantly reduces the number of servers, lowers deployment and maintenance costs, and provides efficient technical support for port automation management and intelligent scheduling. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of an embedded port edge recognition method according to this application;

[0054] Figure 2 This is a schematic diagram of module connections according to one embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the signal flow of one embodiment of this application;

[0056] Figure 4 This is a hardware architecture diagram of an electronic device in this application. Detailed Implementation

[0057] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0058] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this application, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0060] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0061] Additionally, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that practice can be carried out without these specific details.

[0062] In the daily operation of the port, it is necessary to quickly and accurately confirm the identity and status of container trucks entering and leaving the port and the containers they carry, such as: license plate number, container number, dangerous goods label (hazardous goods label), integrity of lead seal, and damage to the container body.

[0063] The current traditional approach involves deploying multi-angle high-definition cameras (front / rear / left / right / top) at the work site (such as crossings and storage yards), combined with inductive loops or video analysis triggers to capture targets; and configuring industrial-grade servers or algorithm industrial control computers in the computer room, relying on their CPU / GPU computing power to run OCR recognition algorithms.

[0064] In view of this, the inventors, through research and improvement exploration, discovered that: existing solutions concentrate computing power on industrial-grade servers or algorithm industrial control computers. On the one hand, a single industrial-grade server or algorithm industrial control computer is expensive, bulky, and consumes a lot of power. If multiple units need to be deployed simultaneously, the hardware cost increases exponentially. On the other hand, the recognition model runs on the PC, and the server CPU / GPU runs a general model. A single recognition takes more than 1 second, causing vehicles to queue and making it difficult to meet the real-time control scenarios such as gate linkage. On the other hand, traditional solutions, which are based on image capture recognition, have high requirements for image capture. In addition, the general OCR model has insufficient recognition accuracy for special targets in ports (such as tilted container numbers, dense dangerous goods signs, etc.), which easily leads to missed detections or false detections.

[0065] Based on this, the embodiments of this specification propose an embedded port edge recognition method: The overall idea is to directly receive the video stream from the port industrial camera, filter out key frames through adaptive frame sampling, and then send them into a customized YOLOv8 model: a scale-compensated feature pyramid is introduced into the Neck, a WIoU v3 loss function is added, and the detection box is changed to a rotated polygon, which can accurately capture multiple types of targets such as license plates, container numbers, dangerous goods signs, lead seals, and damage. Then, the data is divided into categories: character-based targets are parsed by the license plate recognition model, and status-based targets are directly classified by the detection head. The recognition results and location coordinates are transmitted back to the port control system in real time, thereby achieving low-power, serverless fast recognition with an accuracy of over 98% within 200 ms, which meets the needs of various port application scenarios.

[0066] The technical solutions provided by the various embodiments of this application are described below with reference to the accompanying drawings.

[0067] like Figure 1 As shown in the embodiments of this specification, an embedded port edge recognition method is provided, which is executed on an embedded edge device deployed at a port site, including:

[0068] Receive port operation video streams captured by industrial cameras, the video streams including: trucks and containers;

[0069] The video stream is converted into image frames and stored in the shared memory area;

[0070] Based on the adaptive frame sampling algorithm, key frames are selected from the consecutive image frames;

[0071] The keyframes are input into the improved target detection model to obtain the target detection regions annotated with rotated polygon boxes, and the category labels corresponding to each target detection region;

[0072] The improved target detection model is constructed in the following way:

[0073] The backbone network is based on the YOLOv8 architecture;

[0074] Add the WIoU v3 loss function to dynamically focus on hard samples;

[0075] Integrating Scale Compensated Feature Pyramid (SCFPN) in the Neck layer to enhance multi-scale feature fusion;

[0076] The output layer supports rotating polygon frames to adapt to oblique angle shots of containers;

[0077] The category labels include: license plate, container number, dangerous goods label, lead seal, and damage;

[0078] The target detection area is identified. If the category label is a license plate, container number, or dangerous goods label, the license plate recognition model is called to perform character or pattern recognition on the target detection area. If the category label is a lead seal or damage, the improved target detection model is used to classify and identify the target detection area.

[0079] The identification results and corresponding location coordinates are output to the port control system.

[0080] In practice, real-time video streams (H.264 / H.265 encoded) from high-definition cameras (such as 2-megapixel industrial cameras) are received via a gigabit Ethernet interface. These video streams can be parsed into YUV frames by a hardware decoding module such as the VPU of the RK3588J processor and directly stored in the shared memory area, thereby reducing CPU copy overhead.

[0081] Taking port gate recognition as an example, four 2-megapixel industrial cameras are deployed in front of and behind the gate. The specific configuration includes:

[0082] One license plate recognition camera is installed on a column 3 meters in front of the gate, using a 6mm focal length lens;

[0083] Two container side cameras are installed on brackets 2.5 meters high on the left and right sides of the crossing, respectively, and each uses an 8mm focal length lens.

[0084] One top-mounted monitoring camera is installed 4 meters above the level crossing beam, using a 12mm focal length lens.

[0085] When a vehicle enters the recognition area, the video stream is transmitted to the edge terminal via Gigabit Ethernet, converted to YUV420 format by VPU hardware decoding, with a processing latency of less than 50ms. Target detection uses an improved YOLOv8 model, which adds a Scale Compensated Feature Pyramid (SCFPN) module to the Neck layer to address the scale difference between near and far views of the container. It also uses the WIoUv3 loss function, supports outputting rotated polygonal bounding boxes, and is adapted to oblique angle shots of containers. It can detect target areas and corresponding category labels for targets such as license plate numbers, container numbers, container types, hazard markings, damage, and seals in real time.

[0086] For the detected target region, further identification is performed based on the category label:

[0087] If it is a vehicle number, container number, container type, or hazardous material label, then the license plate recognition model (PLRNet model) is used to perform character or pattern recognition on the target detection area, and the recognition result, location box, and confidence score are output.

[0088] If the object is damaged or has a lead seal, the improved target detection (YOLOv8) model is used to classify and identify the target detection area, and output the lead seal type (such as wire-shaped and bullet-shaped) or damage type (such as holes, scratches, rust, oil stains or dents and deformations, etc.) as well as the location box and confidence score.

[0089] Finally, the identification results and corresponding location coordinates are transmitted to the port PLC system in real time to control the gate switch or the hoisting equipment to perform corresponding operations.

[0090] The multi-scale feature enhancement module includes the following steps:

[0091] Step 1, Multi-scale Feature Extraction: Input the multi-level feature maps (C3, C4, C5) output by YOLO8 Backbone. For each level of feature map, extract the following features: Convolution is performed, and the number of channels is adjusted to a preset value (such as 128, 256, 512) to unify the feature dimension. The adjusted feature map is retained as the input for lateral connection and used for subsequent cross-scale feature fusion.

[0092] Step 2, cross-scale feature fusion, adopting a top-down fusion path, specifically:

[0093] Step 21, Top-level feature processing: Using the highest feature map (C5) as the initial fusion feature, through... Convolution extracts semantic information;

[0094] Step 22, layer-by-layer upsampling and fusion, including:

[0095] Step 2201: Upsample the current layer feature map by 2 times (e.g., by nearest neighbor interpolation) to restore spatial resolution;

[0096] Step 2202: Add the upsampled feature map to the next level of horizontally connected feature map (such as the one adjusted by C4) element by element;

[0097] Repeat steps 2201 to 2202 until fused to the lowest feature map (features adjusted by C3).

[0098] Step 23: Generate an enhanced feature map sequence containing multi-scale information;

[0099] Step 3, scale compensation mechanism, specifically:

[0100] A learnable scale compensation factor (scalar) is assigned to each level of enhanced feature map, with an initial value of 1, which is automatically optimized through backpropagation;

[0101] Adjusting dynamic weights: Multiply the scale compensation factor element-wise with the feature map. If it is a small target in the distance, increase the weight of the high-level feature map to compensate for the loss of details; if it is a large target in the foreground, increase the weight of the low-level feature map to preserve spatial information; output the compensated feature map sequence, which significantly enhances scale adaptability.

[0102] Step 4: For each compensated feature map, through... Convolution smooths the feature distribution and reduces sampling aliasing. The processed feature map sequence is input into the YOLO v8 detection head, and the output detection results cover container targets at all scales. The detection results include: class probability and bounding box coordinates.

[0103] In some embodiments, the step of filtering the image frames based on the adaptive frame sampling algorithm includes:

[0104] Set the first image frame of the video stream as the initial keyframe and extract feature points;

[0105] Perform the following checks on each subsequent image frame:

[0106] The optical flow method tracks the position of feature points from the previous keyframe in the current frame frame by frame. If the proportion of successfully tracked feature points is lower than the preset tracking rate threshold, the processing of the current frame is terminated.

[0107] Based on the successfully tracked feature points, the mean optical flow vector and covariance matrix of all successfully tracked feature points are obtained. If the trace of the covariance matrix is ​​greater than a preset motion stability threshold, the current frame processing is terminated.

[0108] The proportion of optical flow anomalies is counted. If the proportion of anomalies is greater than the preset occlusion tolerance threshold, the processing of the current frame is terminated.

[0109] Calculate the Laplacian operator variance of the current frame. If the Laplacian operator variance is less than the preset sharpness threshold, terminate the processing of the current frame.

[0110] If the current frame passes all the criteria, set the current frame as the new keyframe and update the feature points.

[0111] Specifically, first define the keyframe conditions:

[0112] Motion stability: The variance of the global motion vector calculated by the optical flow method is less than the motion stability threshold. ;

[0113] Occlusion detection: The proportion of abnormal points (such as sudden changes in motion direction) in the optical flow field is lower than the occlusion tolerance threshold. ;

[0114] Sharpness: The variance of the Laplacian operator is greater than the sharpness threshold. .

[0115] Read the first frame: Use the first frame of the video as the initial keyframe. And extract feature points (such as ORB feature points).

[0116] Each subsequent image frame is processed frame by frame:

[0117] Input video stream current frame Previous keyframe and its feature points .

[0118] 1. Feature point tracking using optical flow: The Lucas-Kanade optical flow method is used to calculate the keyframe feature points of the garment. In the current frame Corresponding points in The proportion of feature points successfully tracked If the proportion of successfully tracked feature points is lower than the preset tracking rate threshold Then skip the current frame. ,for example: ;

[0119] 2. Calculate global motion consistency: For successfully tracked feature points, calculate their optical flow vectors. Then, calculate the mean of all optical flow vectors. Covariance Matrix If the trace of the covariance matrix If the motion is inconsistent (e.g., occlusion or jitter), then skip the current frame. ;

[0120] 3. Occlusion Detection: Detects outliers in the optical flow field (such as points whose direction of motion is opposite to the mean). If the proportion of outliers is... If it is determined to be an occlusion, the current frame is skipped. ;

[0121] 4. Sharpness Verification: Calculate the current frame. variance of the Laplace operator ,like This indicates the image is blurry; skip the current frame. .

[0122] If the current frame If all the above checks pass, then set it as the new keyframe. And extract its feature points Used for the next round of tracking.

[0123] It should be noted that:

[0124] Motion stability threshold It can be dynamically adjusted based on the motion variance of historical frames, for example: ,in, The smoothing coefficient can be set to 0.8.

[0125] Occlusion tolerance threshold The value can be preset according to the complexity of the scene. For example, the value for the container scene can be preset to 0.1.

[0126] In some embodiments, the embedded port edge recognition method further includes:

[0127] When a container truck enters the inspection area, an industrial camera is triggered by a photoelectric sensor to capture images of the truck and / or container, thus obtaining keyframes.

[0128] The keyframes are stored in the shared memory area after being hardware decoded.

[0129] In practice, the photoelectric sensor triggers the terminal to capture key frames through the General Purpose Input / Output (GPIO) port, and the low hardware latency (<10ms) ensures that the moving target is not blurred.

[0130] In some embodiments, the target detection model and the license plate recognition model are deployed on an NPU processor.

[0131] In some embodiments, the target detection model is converted into an INT8-quantized RKNN format model;

[0132] And / or, convert the license plate recognition model into an INT8 quantized RKNN format model.

[0133] Specifically, YOLOv8 and / or LPRNet models are converted into INT8 quantized RKNN format models, enabling them to run efficiently on the NPU of embedded devices (such as RK3588J). This conversion utilizes the heterogeneous computing capabilities of the NPU (Neural Processing Unit) to keep power consumption below 1.5W and recognition time less than or equal to 200ms, which not only significantly improves the inference speed of the model but also greatly reduces the overall energy consumption of the system.

[0134] In some embodiments, the target detection region is parallelized using an OpenMP-accelerated C++ post-processor before being identified, including performing geometric correction and nonmaximum suppression filtering on the rotated polygonal bounding box.

[0135] In implementation, CPU post-processing is adopted. Specifically, C++ is used to parallelize polygon frame correction and NMS filtering (accelerated by OpenMP) to ensure that the entire processing time is controlled within 50ms.

[0136] Based on the same inventive concept, this application also provides an embedded edge recognition terminal for implementing the aforementioned embedded port edge recognition method, the embedded edge recognition terminal comprising:

[0137] The communication interface unit includes a gigabit Ethernet interface and an RS485 interface. The gigabit Ethernet interface is used to receive port operation video streams (H.264 / H.265 encoded) captured by an industrial camera (e.g., a 2-megapixel camera), the video streams including trucks and containers. The RS485 interface is used to transmit the identification results (e.g., container number) and corresponding location coordinates to the port control system for triggering operations such as gate opening and closing.

[0138] The processing unit adopts an industrial-grade domestic processor chip (such as RK3588J), which integrates: a hardware decoding module, an NPU processor, and a CPU processor.

[0139] The hardware decoding module is used to convert the video stream into YUV image frames and store them in the shared memory area to reduce CPU copy overhead. The NPU processor is used to deploy the improved object detection model (YOLOv8) and license plate recognition model (PLRNet), which run on the NPU in INT8 quantization format, with a single inference time of 35ms, and supports special formats such as new energy vehicle license plates. The CPU processor is used to run the Linux operating system, which can run for a long time to ensure stable system operation.

[0140] A customized circuit board, which is matched with the processing unit, includes a multi-layer PCB board (e.g., a 4-layer PCB board); the customized circuit board is used to carry the communication interface unit, the processing unit, and the memory of the storage unit.

[0141] In practice, by using an adaptive frame sampling algorithm (based on optical flow motion estimation) to filter key frames from the video stream (such as when the front of a container is unobstructed), the recognition rate is improved by 23% compared to fixed frame rate capture.

[0142] Under typical port environments (-20℃~60℃), the overall power consumption of the terminal is ≤15W, and it can operate without failure 24 / 7.

[0143] In some embodiments, the embedded edge recognition terminal further includes: a dual gigabit PHY chip and an isolation circuit;

[0144] The dual gigabit PHY chip is connected to the processing unit to form a redundant second gigabit Ethernet interface to support automatic switching between fiber optic and copper cables.

[0145] The isolation circuit is used to block electromagnetic interference in the port.

[0146] In practice, through redundancy design, when one link (fiber optic cable or network port) fails, the system can automatically switch to another link to ensure uninterrupted communication.

[0147] In addition, there are a large number of electrical and mechanical devices in the port environment, such as cranes, loading and unloading equipment, ship engines, and high-voltage power lines. These devices generate strong electromagnetic fields during operation, which can cause electromagnetic interference. Adding isolation circuits can isolate sensitive electronic equipment from electromagnetic interference sources in the port environment, thereby protecting the normal operation of the equipment and significantly improving its reliability and stability.

[0148] Based on the same inventive concept, such as Figure 2 As shown, this application also provides a port operation control system, which includes:

[0149] At least one industrial camera 1, gigabit network switch 2, the aforementioned embedded edge recognition terminal 3, and PLC controller;

[0150] The industrial camera 1 is deployed at level crossings, rail-mounted gantry cranes, and / or bridge cranes. The industrial camera is used to collect real-time video streams of trucks and containers.

[0151] The gigabit network switch 2 is used to connect the industrial camera 1 to the embedded edge recognition terminal 3 via gigabit Ethernet;

[0152] The embedded edge recognition terminal 3 is used to recognize the video stream and output the recognition result and the corresponding position coordinates;

[0153] The PLC controller receives the recognition result and corresponding position coordinates from the embedded edge recognition terminal, and controls the barrier gate or hoisting equipment to perform corresponding operations based on the recognition result.

[0154] Specifically, such as Figure 2 and 3 As shown, industrial camera 1, serving as image / video stream input A, is deployed at key locations in the port, such as level crossings, rail-mounted gantry cranes, or bridge cranes, to capture images or videos of trucks and containers. Gigabit network switch 2 transmits the video stream captured by industrial camera 1 to the embedded edge recognition terminal 3 via gigabit Ethernet. It should be noted that there can be multiple industrial cameras 1, such as... Figure 2 As shown in 1a, 1b, 1c, 1d, and 1e, there can be multiple edge recognition terminals 3, such as... Figure 2 As shown in 3a and 3b.

[0155] Inside the embedded edge recognition terminal, the video stream is converted into image frames by a hardware decoding module and stored in the shared memory area. The OCR recognition algorithm is run in the processing unit (such as the NPU processor) to identify information such as license plates and container numbers, forming the processing link C. Subsequently, the embedded edge recognition terminal 3 forwards these recognition results to the PLC controller as B through the RS485 interface or other communication methods, which is used to control the gate switch or hoisting equipment to perform corresponding operations, realizing the "acquisition-recognition-control" closed loop.

[0156] The following is another example, which combines the previous examples to form an illustrative example.

[0157] Taking the crane identification scenario as an example, intelligent identification of container and vehicle information is achieved. Based on the number of lanes below the crane, multiple high-definition industrial cameras are deployed on the top and crossbeams of the crane, with the specific configuration as follows:

[0158] Hardware deployment:

[0159] Top camera system: A varying number of high-definition cameras are installed on the top of the gantry crane and crossbeams, depending on the number of lanes below, to ensure complete coverage of the gantry crane's operating area. The main cameras are responsible for collecting: truck top number information; container number and type identification; and information on container top damage.

[0160] Edge computing terminal: Each bridge crane is equipped with one ruggedized edge recognition terminal, with core configurations including:

[0161] Processor: Rockchip RK3588J industrial-grade chip.

[0162] Recognition algorithm process:

[0163] (1) Image acquisition stage:

[0164] The top camera shoots continuously at 15fps.

[0165] The beam camera triggers a snapshot when the spreader moves above the container.

[0166] All images are transmitted to the edge terminal in real time via gigabit Ethernet.

[0167] (2) Target detection stage:

[0168] Multi-object detection using an improved YOLOv8s model:

[0169] The detection targets include:

[0170] Roof numbering area (rotating rectangle)

[0171] Marking areas (polygonal frames) on each side of the container.

[0172] Hazardous area (rectangular frame)

[0173] Detection accuracy: 98.7% at mAP@0.5;

[0174] (3) OCR recognition stage:

[0175] Roof number recognition: Using the PLRNet model, it supports recognition at tilt angles (±45°) with an accuracy rate of ≥99.2%.

[0176] The following is another example, which combines the previous examples to form an illustrative example.

[0177] Taking the rail-mounted crane scene recognition as an example, the system is equipped with six 5-megapixel global shutter cameras, evenly mounted on the lifting device support with a spacing of 1.2 meters. Each camera is equipped with a laser rangefinder for dynamically adjusting the focal length. To address vibration issues during lifting operations, rubber shock-absorbing pads are added to the camera mounting bases, and motion blur compensation technology based on Wiener filtering is introduced into the image processing algorithm.

[0178] This application uses an independently developed embedded edge recognition terminal, which is convenient to deploy in port scenarios for recognition, and is low in cost, small in size, and low in power consumption.

[0179] The OCR recognition algorithm in this application not only supports image capture recognition, but also video stream recognition, that is, continuous recognition using the video stream of the camera during the movement of containers or vehicles. In this way, the algorithm can automatically acquire better image frames in the video stream for continuous recognition, ensuring a higher recognition rate and accuracy.

[0180] This embodiment provides an electronic device, which can be represented in the form of a computing device (e.g., a server device), including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it can implement the steps of the embedded port edge recognition method provided in the above embodiment.

[0181] Figure 4 A schematic diagram of the hardware structure of this embodiment is shown, as follows: Figure 4 As shown, the electronic device 30 specifically includes:

[0182] At least one processor 31, at least one memory 32, and a bus 33 for connecting different system components (including the processor 31 and the memory 32), wherein:

[0183] Bus 33 includes a data bus, an address bus, and a control bus.

[0184] The memory 32 includes volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0185] The memory 32 also includes a program / utility 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0186] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the steps of the embedded port edge recognition method provided in the above embodiments of the present invention.

[0187] Electronic device 30 can further communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 35. Furthermore, electronic device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 36. Network adapter 36 communicates with other modules of electronic device 30 via bus 33. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0188] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0189] In this specification, the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the descriptions of the embodiments described later are relatively simple, and relevant parts can be referred to the descriptions of the foregoing embodiments.

[0190] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An embedded port edge recognition method, characterized in that, Executed on embedded edge devices deployed at the port site, including: Receive port operation video streams captured by industrial cameras, the video streams including: trucks and containers; The video stream is converted into image frames and stored in the shared memory area; Based on the adaptive frame sampling algorithm, key frames are selected from the consecutive image frames; The keyframes are input into the improved target detection model to obtain the target detection regions annotated with rotated polygon boxes, and the category labels corresponding to each target detection region; The improved target detection model is constructed in the following way: The backbone network is based on the YOLOv8 architecture; Add the WIoU v3 loss function to dynamically focus on hard samples; Integrate a scale-compensated feature pyramid in the Neck layer to enhance multi-scale feature fusion; The output layer supports rotating polygon frames to adapt to oblique angle shots of containers; The category labels include: license plate, container number, dangerous goods label, lead seal, and damage; The target detection area is identified. If the category label is a license plate, container number, or dangerous goods label, the license plate recognition model is called to perform character or pattern recognition on the target detection area. If the category label is a lead seal or damage, the improved target detection model is used to classify and identify the target detection area. The identification results and corresponding location coordinates are output to the port control system.

2. The embedded port edge recognition method according to claim 1, characterized in that, The step of filtering the image frames based on the adaptive frame sampling algorithm includes: Set the first image frame of the video stream as the initial keyframe and extract feature points; Perform the following checks on each subsequent image frame: The optical flow method tracks the position of feature points from the previous keyframe in the current frame frame by frame. If the proportion of successfully tracked feature points is lower than the preset tracking rate threshold, the processing of the current frame is terminated. Based on the successfully tracked feature points, the mean optical flow vector and covariance matrix of all successfully tracked feature points are obtained. If the trace of the covariance matrix is ​​greater than a preset motion stability threshold, the current frame processing is terminated. The proportion of optical flow anomalies is counted. If the proportion of anomalies is greater than the preset occlusion tolerance threshold, the processing of the current frame is terminated. Calculate the Laplacian operator variance of the current frame. If the Laplacian operator variance is less than the preset sharpness threshold, terminate the processing of the current frame. If the current frame passes all the criteria, set the current frame as the new keyframe and update the feature points.

3. The embedded port edge recognition method according to claim 1, characterized in that, The embedded port edge recognition method further includes: When a container truck enters the inspection area, an industrial camera is triggered by a photoelectric sensor to capture images of the truck and / or container, thus obtaining keyframes. The keyframes are stored in the shared memory area after being hardware decoded.

4. The embedded port edge recognition method according to claim 1, characterized in that, The target detection model and the license plate recognition model are deployed on the NPU processor.

5. The embedded port edge recognition method according to claim 4, characterized in that, The target detection model is converted into an INT8-quantized RKNN format model; And / or, convert the license plate recognition model into an INT8 quantized RKNN format model.

6. The embedded port edge recognition method according to claim 1, characterized in that, Before identifying the target detection region, the target detection region is processed in parallel using an OpenMP-accelerated C++ post-processor, including performing geometric correction and non-maximum suppression filtering on the rotated polygonal box.

7. An embedded edge recognition terminal, characterized in that, For implementing the embedded port edge recognition method as described in any one of claims 1-6, the embedded edge recognition terminal includes: The communication interface unit includes a gigabit Ethernet interface and an RS485 interface; wherein, the gigabit Ethernet interface is used to receive port operation video streams captured by industrial cameras, the video streams including trucks and containers; the RS485 interface is used to transmit the identification results and corresponding location coordinates to the port control system. The processing unit adopts an industrial-grade domestic processor chip, which integrates a hardware decoding module, an NPU processor, and a CPU processor. The hardware decoding module is used to convert the video stream into image frames and store them in a shared memory area; the NPU processor is used to deploy an improved target detection model and license plate recognition model; and the CPU processor is used to run the Linux operating system. A customized circuit board, which is matched with the processing unit, includes a multi-layer PCB board; the customized circuit board is used to carry the communication interface unit, the processing unit, and the memory of the storage unit.

8. The embedded edge recognition terminal according to claim 7, characterized in that, The embedded edge recognition terminal also includes: dual gigabit PHY chips and isolation circuitry; The dual gigabit PHY chip is connected to the processing unit to form a redundant second gigabit Ethernet interface to support automatic switching between fiber optic and copper cables. The isolation circuit is used to block electromagnetic interference in the port.

9. A port operation control system, characterized in that, The port operation control system includes: At least one industrial camera, a gigabit network switch, an embedded edge recognition terminal as described in any one of claims 7-8, and a PLC controller; The industrial cameras are deployed at level crossings, rail-mounted gantry cranes, and / or gantry cranes to capture real-time video streams from trucks and containers; The gigabit network switch is used to connect the industrial camera to the embedded edge recognition terminal via gigabit Ethernet; The embedded edge recognition terminal is used to recognize the video stream and output the recognition result and the corresponding position coordinates; The PLC controller receives the recognition result and corresponding position coordinates from the embedded edge recognition terminal, and controls the barrier gate or hoisting equipment to perform corresponding operations based on the recognition result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the embedded port edge recognition method as described in any one of claims 1-6.

Images