Embedded port edge intelligent identification method, identification terminal, equipment and control system

Abstract

The invention provides an embedded port edge intelligent identification method, an identification terminal, equipment and a control system, and is applied to the technical field of identification of basic information of license plates and containers of port services. Port operation video streams collected by an industrial camera are received, the video streams are converted into image frames, and key frames are screened out; an improved YOLOv8 target detection model is used for carrying out high-precision multi-target detection, corresponding recognition models are called for different types of labels, efficient and accurate recognition of license plates, container numbers, dangerous labels, lead seals, damage and other information of container trucks and containers in a port scene is achieved, the overall recognition time delay is shortened to be within 200 ms, the accuracy rate is increased to 98% or above, and the recognition efficiency is improved. The number of servers is remarkably reduced, the deployment and operation and maintenance cost is reduced, and efficient technical support is provided for automatic management and intelligent scheduling of ports.

Description

Technical Field

[0001] The present application relates to the technical field of license plate and container basic information recognition for port business, and specifically to an embedded port edge intelligent recognition method, recognition terminal, equipment and control system. Background Art

[0002] At present, the software and hardware technical solutions for port OCR recognition mainly adopt a centralized architecture: multiple high-definition network cameras (deployed at multiple angles, including front, back, left, right, and center), ground sensor coils, or video analysis triggers are installed at the recognition site to assist in image acquisition, and industrial-grade servers or algorithm-based industrial computers are deployed in the computer room. OCR recognition algorithms are configured on these servers or algorithm-based industrial computers, and the OCR recognition algorithms are run using the server CPU or the computing power card / graphics card of the algorithm-based industrial computer.

[0003] However, since these recognition models run on PCs, the recognition speed is relatively slow, usually exceeding one second per recognition, which cannot meet the fast-paced requirements of continuous port traffic or crane operations. Moreover, traditional solutions are based on image capture recognition, which has high requirements for the quality of the captured images, and is prone to high missed detection and false detection rates in actual operations. In addition, industrial-grade servers or algorithm-based industrial computers are expensive, large in size, and have high power consumption. If port operations require the simultaneous deployment of multiple such devices, the overall cost will increase significantly.

[0004] Based on this, a new port OCR recognition solution is needed. Summary of the Invention

[0005] In view of this, the embodiments of this specification provide an embedded port edge intelligent recognition method, recognition terminal, equipment and control system, which achieve millisecond-level recognition of port vehicles and containers through an improved target detection model and license plate recognition model.

[0006] The embodiments of this specification provide the following technical solutions: In a first aspect, embodiments of this specification provide an embedded port edge recognition method, which is executed on an embedded edge device deployed at a port site, comprising: Receive a port operation video stream captured by an industrial camera, the video stream including: container trucks and containers; Convert the video stream into image frames and store them in a shared memory area; Based on an adaptive frame sampling algorithm, key frames are selected from the continuous image frames; Inputting the key frame into an improved target detection model to obtain target detection areas marked by rotating polygonal boxes and category labels corresponding to each target detection area; The improved target detection model is constructed in the following way: The backbone network is based on the YOLOv8 architecture; Add WIoU v3 loss function to dynamically focus on difficult samples; Integrate scale-compensated feature pyramids in the Neck layer to enhance multi-scale feature fusion; The output layer supports rotating polygons to adapt to oblique angles of the container. The categories of labels include: license plate, container number, danger label, seal and damage; Identify the target detection area. If the category label is a license plate, container number, or danger sign, call the license plate recognition model to perform character or pattern recognition on the target detection area. If the category label is a seal or damage, use the improved target detection model to classify and identify the target detection area. Output the recognition results and corresponding location coordinates to the port control system.

[0007] Furthermore, the screening of the image frames based on the adaptive frame sampling algorithm includes: Setting the first image frame of the video stream as an initial key frame and extracting feature points; For each subsequent image frame, the following checks are performed: Based on the optical flow method, the position of the feature points of the previous key frame in the current frame is tracked frame by frame. If the proportion of successfully tracked feature points is lower than the preset tracking rate threshold, the current frame processing is terminated; Based on the successfully tracked feature points, the optical flow vector mean and covariance matrix of all successfully tracked feature points are obtained, and if the trace of the covariance matrix is ​​greater than a preset motion stability threshold, the current frame processing is terminated; Count the proportion of optical flow abnormal points. If the proportion of abnormal points is greater than the preset occlusion tolerance threshold, terminate the current frame processing; Calculate the Laplace operator variance of the current frame. If the Laplace operator variance is less than the preset clarity threshold, terminate the current frame processing. If the current frame passes all judgments, the current frame is set as the new key frame and the feature points are updated.

[0008] Furthermore, the embedded port edge recognition method further includes: When a container truck enters the detection area, a photoelectric sensor is used to trigger an industrial camera to capture images of the truck and / or container to obtain key frames. The key frame is stored in the memory shared area after hard decoding.

[0009] Furthermore, the target detection model and the license plate recognition model are deployed on an NPU processor.

[0010] Furthermore, the target detection model is converted into an INT8 quantized RKNN format model; And / or, converting the license plate recognition model into an INT8 quantized RKNN format model.

[0011] Furthermore, before identifying the target detection area, a C++ post-processing program accelerated by OpenMP is used to perform parallel processing on the target detection area, including: performing geometric correction and non-maximum suppression filtering on the rotated polygon box.

[0012] In a second aspect, an embodiment of this specification further provides an embedded edge identification terminal for implementing the aforementioned embedded port edge identification method, wherein the embedded edge identification terminal includes: A communication interface unit comprising a Gigabit Ethernet interface and an RS485 interface; wherein the Gigabit Ethernet interface is used to receive a video stream of port operations captured by an industrial camera, the video stream including trucks and containers; and the RS485 interface is used to transmit the recognition results and corresponding location coordinates to the port control system; Processing unit, the processing unit adopts industrial-grade domestic processor chip, the industrial-grade domestic processor chip integrates: hard decoding module, NPU processor and CPU processor; The hardware decoding module is used to convert the video stream into image frames and store them in a shared memory area; the NPU processor is used to deploy an improved target detection model and license plate recognition model; and the CPU processor is used to run a Linux operating system. A customized circuit board, which is matched with the processing unit, includes a multi-layer PCB board; the customized circuit board is used to carry the communication interface unit, the processing unit and the storage unit memory.

[0013] Furthermore, the embedded edge identification terminal further comprises: a dual gigabit PHY chip and an isolation circuit; The dual Gigabit PHY chip is connected to the processing unit to form a redundant second Gigabit Ethernet interface to support automatic switching of optical fiber and copper cable; The isolated circuit is used to block electromagnetic interference in the port.

[0014] In a third aspect, the embodiments of this specification further provide a port operation control system, the port operation control system comprising: At least one industrial camera, a gigabit network switch, the aforementioned embedded edge recognition terminal, and a PLC controller; The industrial camera is deployed at a road crossing, rail crane and / or bridge crane, and is used to collect real-time video streams of container trucks and containers; The gigabit network switch is used to connect the industrial camera to the embedded edge recognition terminal via gigabit Ethernet; The embedded edge recognition terminal is used to recognize the video stream and output the recognition result and the corresponding position coordinates; The PLC controller receives the recognition result and corresponding position coordinates of the embedded edge recognition terminal, and controls the gate or lifting equipment to perform corresponding operations according to the recognition result.

[0015] In a fourth aspect, an embodiment of this specification also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned embedded port edge recognition method when executing the computer program.

[0016] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: By receiving the port operation video stream captured by industrial cameras, converting the video stream into image frames and filtering out key frames, the improved YOLOv8 target detection model is used to perform high-precision multi-target detection, and the corresponding recognition model is called for different category labels. It achieves efficient and accurate recognition of license plates, box numbers, danger labels, seals and damage information of container trucks and containers in port scenes, reduces the overall recognition delay to less than 200 ms, and improves the accuracy rate to more than 98%. It significantly reduces the number of servers, reduces deployment and operation and maintenance costs, and provides efficient technical support for port automation management and intelligent scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 is a flow chart of an embedded port edge identification method in this application; Figure 2 This is a schematic diagram of module connections in one embodiment of the present application; Figure 3 This is a signal flow diagram of an embodiment of the present application; Figure 4 This is a hardware architecture diagram of an electronic device in this application. DETAILED DESCRIPTION

[0019] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0020] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the features in the following embodiments and embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0021] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on this application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement an apparatus and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this apparatus and / or practice this method.

[0022] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show components related to the present application and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component can be changed at will, and the component layout type may also be more complicated.

[0023] Additionally, in the following description, specific details are provided to provide a thorough understanding of the examples, however, one skilled in the art will appreciate that the examples can be practiced without these specific details.

[0024] In daily port operations, it is necessary to quickly and accurately confirm the identity and status of container trucks entering and leaving the port and the containers they carry, such as: license plate number, container number, hazardous materials identification (hazardous goods label), lead seal integrity, and container damage.

[0025] The traditional solution is to deploy multi-angle high-definition cameras (front / back / left / right / top) at the work site (such as a road crossing or storage yard), combined with ground sensor coils or video analysis triggers to capture targets; and configure industrial-grade servers or algorithm-based industrial computers in the computer room, relying on their CPU / GPU computing power to run the OCR recognition algorithm.

[0026] In view of this, the inventors discovered through research and improvement exploration that the existing solutions concentrate computing power on industrial-grade servers or algorithm-based industrial computers. On the one hand, an industrial-grade server or algorithm-based industrial computer is costly, bulky, and consumes high power. If multiple servers need to be deployed at the same time, the hardware cost will increase exponentially. On the other hand, the recognition model runs on the PC side, and the server CPU / GPU runs a general model. A single recognition takes more than 1 second, resulting in vehicle queues, making it difficult to meet real-time control scenarios such as gate linkage. On the third hand, the traditional solution is based on image capture-based recognition, which has relatively high requirements for image capture. In addition, the general OCR model has insufficient recognition accuracy for special targets in ports (such as tilted container numbers, dense dangerous goods signs, etc.), which can easily lead to missed detections or false detections.

[0027] Based on this, the embodiment of this specification proposes an embedded port edge recognition method: the overall idea is: directly receive the video stream of the port industrial camera, filter out the key frames through adaptive frame sampling, and send it to the customized YOLOv8 model: introduce the scale compensation feature pyramid in Neck, add the WIoU v3 loss function, and change the detection frame to a rotated polygon, which can accurately capture multiple types of targets such as license plates, box numbers, dangerous goods labels, lead seals, and damage, and then divert them by category. The character class is parsed by the license plate recognition model, and the state class is directly classified by the detection head, and the recognition results and position coordinates are sent back to the port control system in real time, thereby achieving low-power, server-free and fast recognition with an accuracy rate of more than 98% within 200 ms, meeting various application scenarios of the port.

[0028] The following describes the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0029] like Figure 1 As shown, the embodiment of this specification provides an embedded port edge recognition method, which is executed on an embedded edge device deployed at a port site, including: Receive a port operation video stream captured by an industrial camera, the video stream including: container trucks and containers; Convert the video stream into image frames and store them in a shared memory area; Based on an adaptive frame sampling algorithm, key frames are selected from the continuous image frames; Inputting the key frame into an improved target detection model to obtain target detection areas marked by rotating polygonal boxes and category labels corresponding to each target detection area; The improved target detection model is constructed in the following way: The backbone network is based on the YOLOv8 architecture; Add WIoU v3 loss function to dynamically focus on difficult samples; Integrate scale-compensated feature pyramid (SCFPN) in the Neck layer to enhance multi-scale feature fusion; The output layer supports rotating polygons to adapt to oblique angles of the container. The categories of labels include: license plate, container number, danger label, seal and damage; Identify the target detection area. If the category label is a license plate, container number, or danger sign, call the license plate recognition model to perform character or pattern recognition on the target detection area. If the category label is a seal or damage, use the improved target detection model to classify and identify the target detection area. Output the recognition results and corresponding location coordinates to the port control system.

[0030] In implementation, real-time video streams (H.264 / H.265 encoding) from high-definition cameras (such as 2-megapixel industrial cameras) are received through the Gigabit Ethernet interface. These video streams can be parsed into YUV frames by the VPU hard decoding module of the RK3588J processor and directly stored in the shared memory area, thereby reducing CPU copy overhead.

[0031] Taking the port crossing recognition scenario as an example, four 2-megapixel industrial cameras are deployed in front and behind the crossing gate. The specific configuration includes: A license plate recognition camera is installed on a column 3 meters in front of the gate, using a 6mm focal length lens; Two container side cameras are installed on 2.5-meter-high brackets on the left and right sides of the crossing, using 8mm focal length lenses; A top-mounted surveillance camera is installed at a height of 4 meters on the crossing beam and uses a 12mm focal length lens.

[0032] When a vehicle enters the recognition area, the video stream is transmitted to the edge terminal via Gigabit Ethernet and converted to YUV420 format via VPU hardware decoding, with processing latency less than 50ms. Object detection utilizes an improved YOLOv8 model, which adds a scale-compensated feature pyramid (SCFPN) module to the Neck layer to address scale differences between near and far container scenes. The model also employs the WIoUv3 loss function, supports outputting rotated polygonal boxes, and adapts to oblique container angles. This allows for real-time detection of target areas and corresponding category labels for objects such as license plates, container numbers, container types, hazardous material labels, damage labels, and seals.

[0033] For the detected target area, further identification is performed based on the category label: If it is a vehicle number, box number, box type or danger sign, the license plate recognition model (PLRNet model) is used to perform character or pattern recognition on the target detection area, and output the recognition result, location box and confidence level.

[0034] If it is damaged or sealed, the improved target detection (yolo8) model is used to classify and identify the target detection area, and output the seal type (such as wire shape and bullet shape) or damage type (such as holes, scratches, rust, oil stains or concave and convex deformation, etc.) as well as the location box and confidence level.

[0035] Finally, the recognition results and corresponding position coordinates are transmitted to the port PLC system in real time to control the gate switch or lifting equipment to perform corresponding operations.

[0036] Among them, the multi-scale feature enhancement module includes the following steps: Step 1, multi-scale feature extraction: Input the multi-level feature map (C3, C4, C5) output by YOLO8 Backbone, and for each level of feature map, Convolution adjusts the number of channels to a preset value (such as 128, 256, or 512), unifies the feature dimension, and retains the adjusted feature map as the horizontal connection input for subsequent cross-scale feature fusion.

[0037] Step 2: Cross-scale feature fusion, using a top-down fusion path. Specifically: Step 21, top-level feature processing: take the highest feature map (C5) as the initial fusion feature, through Convolution extracts semantic information; Step 22: Upsampling and fusion layer by layer, including: Step 2201: perform a 2x upsampling (e.g., nearest neighbor interpolation) on the feature map of the current level to restore the spatial resolution; Step 2202: Add the upsampled feature map and the next level horizontal connection feature map (such as C4 adjusted feature map) element by element. Repeat steps 2201 to 2202 until the bottom feature map (features adjusted by C3) is fused. Step 23: Generate an enhanced feature map sequence containing multi-scale information; Step 3: Scale compensation mechanism, specifically: Assign a learnable scale compensation factor (scalar) to each level of enhanced feature map, with the initial value set to 1 and automatically optimized through backpropagation; Adjust dynamic weights: Multiply the scale compensation factor by the feature map element by element. If it is a small target in the distance, the weight of the high-level feature map is increased to compensate for the loss of details; if it is a large target in the near distance, the weight of the low-level feature map is increased to preserve spatial information. The compensated feature map sequence is output, and the scale adaptability is significantly enhanced. Step 4: For each level of feature map after compensation, Convolution smoothes the feature distribution and reduces sampling aliasing effects. The processed feature map sequence is input into the YOLO v8 detection head, and the output detection results cover container targets of all scales. The detection results include: category probability and bounding box coordinates.

[0038] In some embodiments, screening the image frames based on an adaptive frame sampling algorithm includes: Setting the first image frame of the video stream as an initial key frame and extracting feature points; For each subsequent image frame, the following checks are performed: Based on the optical flow method, the position of the feature points of the previous key frame in the current frame is tracked frame by frame. If the proportion of successfully tracked feature points is lower than the preset tracking rate threshold, the current frame processing is terminated; Based on the successfully tracked feature points, the optical flow vector mean and covariance matrix of all successfully tracked feature points are obtained, and if the trace of the covariance matrix is ​​greater than a preset motion stability threshold, the current frame processing is terminated; Count the proportion of optical flow abnormal points. If the proportion of abnormal points is greater than the preset occlusion tolerance threshold, terminate the current frame processing; Calculate the Laplace operator variance of the current frame. If the Laplace operator variance is less than the preset clarity threshold, terminate the current frame processing. If the current frame passes all judgments, the current frame is set as the new key frame and the feature points are updated.

[0039] Specifically, first define the key frame conditions: Motion stability: The variance of the global motion vector calculated by the optical flow method is less than the motion stability threshold ; Occlusion detection: The proportion of abnormal points in the optical flow field (such as sudden changes in motion direction) is lower than the occlusion tolerance threshold ; Clarity: Laplace operator variance is greater than the clarity threshold .

[0040] Read first frame: Use the first frame of the video as the initial key frame , and extract feature points (such as ORB feature points).

[0041] Process each subsequent image frame frame by frame: Current frame of the input video stream , previous keyframe and its characteristic points .

[0042] 1. Optical flow method to track feature points: Lucas-Kanade optical flow method is used to calculate the key frame feature points of the top In the current frame The corresponding points in , the proportion of feature points that are successfully tracked If the ratio of successfully tracked feature points is lower than the preset tracking rate threshold , then skip the current frame ,for example: ; 2. Calculate global motion consistency: Calculate the optical flow vector of the successfully tracked feature points , then, calculate the mean of all optical flow vectors and covariance matrix , if the trace of the covariance matrix , indicating inconsistent motion (such as occlusion or shaking), then skip the current frame ; 3. Occlusion detection: Detect abnormal points in the optical flow field (such as points whose movement direction is opposite to the mean). If the abnormal points account for , it is determined to be blocked and the current frame is skipped ; 4. Clarity Verification: Calculate the current frame The Laplace operator variance ,like , indicating that the image is blurred, skip the current frame .

[0043] If the current frame If all the above checks are passed, it will be set as a new keyframe , and extract its feature points Used for the next round of tracking.

[0044] It should be noted that: Motion stability threshold It can be dynamically adjusted based on the motion variance of historical frames, for example: ,in, is the smoothing coefficient, which can be taken as 0.8.

[0045] Occlusion tolerance threshold It can be preset according to the complexity of the scene, for example, the container scene can be preset to 0.1.

[0046] In some embodiments, the embedded port edge identification method further includes: When a container truck enters the detection area, the photoelectric sensor triggers the industrial camera to capture images of the truck and / or container to obtain key frames; The key frame is stored in the shared memory area after hard decoding.

[0047] During implementation, the photoelectric sensor triggers the terminal to capture key frames through the General Purpose Input / Output (GPIO) port, and the hardware-level low latency (<10ms) ensures that the moving target is not blurred.

[0048] In some embodiments, the target detection model and the license plate recognition model are deployed on an NPU processor.

[0049] In some embodiments, the target detection model is converted into an INT8 quantized RKNN format model; And / or, converting the license plate recognition model into an INT8 quantized RKNN format model.

[0050] Specifically, the YOLOv8 and / or LPRNet models are converted into INT8-quantized RKNN format models so that they can run efficiently on the NPU of embedded devices (such as RK3588J). This conversion controls power consumption within 1.5W and recognition time less than or equal to 200ms by leveraging the heterogeneous computing capabilities of the NPU (neural network processing unit). This not only significantly improves the model's inference speed, but also greatly reduces the overall energy consumption of the system.

[0051] In some embodiments, before identifying the target detection area, a C++ post-processing program accelerated by OpenMP is used to perform parallel processing on the target detection area, including: performing geometric correction and non-maximum suppression filtering on the rotated polygon box.

[0052] During implementation, CPU post-processing was adopted. Specifically, C++ was used to parallelize polygon correction and NMS filtering (accelerated by OpenMP) to ensure that the entire processing time was controlled within 50ms.

[0053] Based on the same inventive concept, the present application also provides an embedded edge recognition terminal for implementing the aforementioned embedded port edge recognition method, the embedded edge recognition terminal comprising: A communication interface unit, comprising a Gigabit Ethernet interface and an RS485 interface. The Gigabit Ethernet interface is used to receive a port operation video stream (H.264 / H.265 encoding) captured by an industrial camera (e.g., 2-megapixel), the video stream including trucks and containers. The RS485 interface is used to transmit the identification results (e.g., container number) and corresponding location coordinates to the port control system for triggering operations such as gate opening and closing. Processing unit, the processing unit adopts industrial-grade domestic processor chip (such as RK3588J), the industrial-grade domestic processor chip integrates: hard decoding module, NPU processor and CPU processor; The hardware decoding module is used to convert the video stream into YUV image frames and store them in a shared memory area, reducing CPU copy overhead. The NPU processor is used to deploy an improved object detection model (YOLOv8) and license plate recognition model (PLRNet), which run in INT8 quantization format on the NPU, with a single inference time of 35ms and support for special formats such as new energy vehicle license plates. The CPU processor is used to run the Linux operating system and can run for a long time to ensure stable system operation. A customized circuit board, which is matched with the processing unit, includes a multi-layer PCB board (for example, a 4-layer PCB board); the customized circuit board is used to carry the communication interface unit, the processing unit and the storage unit memory.

[0054] During implementation, an adaptive frame sampling algorithm (based on optical flow motion estimation) is used to select key frames from the video stream (such as when the front of the container is unobstructed), and the recognition rate is increased by 23% compared to fixed frame rate capture.

[0055] In a typical port environment (-20℃~60℃), the terminal's overall power consumption is ≤15W and it operates 24 / 7 without any faults.

[0056] In some embodiments, the embedded edge identification terminal further comprises: a dual gigabit PHY chip and an isolated circuit; The dual Gigabit PHY chip is connected to the processing unit to form a redundant second Gigabit Ethernet interface to support automatic switching of optical fiber and copper cable; The isolated circuit is used to block electromagnetic interference in the port.

[0057] During implementation, through redundant design, when one link (fiber or network port) fails, the system can automatically switch to another link to ensure uninterrupted communication.

[0058] In addition, there are a large number of electrical equipment and mechanical devices in the port environment, such as cranes, loading and unloading equipment, ship engines, high-voltage power lines, etc. These devices will generate strong electromagnetic fields during operation, causing electromagnetic interference. Adding isolated circuits can isolate sensitive electronic equipment from electromagnetic interference sources in the port environment, thereby protecting the normal operation of the equipment and significantly improving the reliability and stability of the equipment.

[0059] Based on the same inventive concept, Figure 2 As shown, the present application also provides a port operation control system, which includes: At least one industrial camera 1, a gigabit network switch 2, the aforementioned embedded edge recognition terminal 3, and a PLC controller; The industrial camera 1 is deployed at a road crossing, rail crane and / or bridge crane, and is used to collect real-time video streams of container trucks and containers; The Gigabit network switch 2 is used to connect the industrial camera 1 to the embedded edge recognition terminal 3 via Gigabit Ethernet; The embedded edge recognition terminal 3 is used to recognize the video stream and output the recognition result and the corresponding position coordinates; The PLC controller receives the recognition result and corresponding position coordinates of the embedded edge recognition terminal, and controls the gate or lifting equipment to perform corresponding operations according to the recognition result.

[0060] Specifically, if Figure 2 and 3 As shown, the industrial camera 1 is used as the image / video stream input A and is deployed at key locations in the port, such as road crossings, rail cranes or bridge cranes, to capture images or videos of container trucks and containers. The Gigabit network switch 2 transmits the video stream captured by the industrial camera 1 to the embedded edge recognition terminal 3 via Gigabit Ethernet. It should be noted that there can be multiple industrial cameras 1, such as Figure 2 As shown in 1a, 1b, 1c, 1d, and 1e, there may be multiple edge recognition terminals 3, such as Figure 2 As shown in 3a and 3b.

[0061] Inside the embedded edge recognition terminal, the video stream is converted into image frames through the hard decoding module and stored in the shared memory area. The OCR recognition algorithm is run on the processing unit (such as the NPU processor) to identify information such as license plates and container numbers, forming processing link C. Subsequently, the embedded edge recognition terminal 3 forwards these recognition results as B to the PLC controller through the RS485 interface or other communication methods, which is used to control the gate switch or lifting equipment to perform corresponding operations, realizing the "acquisition-recognition-control" closed loop.

[0062] Here is another example, which is a schematic illustration formed by combining the above examples.

[0063] Taking the bridge crane recognition scenario as an example, to achieve intelligent recognition of container and vehicle information, multiple high-definition industrial cameras are deployed on the top and crossbeam of the bridge crane according to the number of lanes below the bridge crane. The specific configuration is as follows: Hardware deployment: Top camera system: Different numbers of high-definition cameras are installed on the top and crossbeam of the bridge crane according to the number of lanes below to ensure complete coverage of the bridge crane operation area. They are mainly responsible for collecting: truck roof number information; container number and type identification; and damage to the container top.

[0064] Edge computing terminal: Each bridge crane is equipped with a reinforced edge recognition terminal. The core configuration includes: Processor: Rockchip RK3588J industrial-grade chip.

[0065] Recognition algorithm process: (1) Image acquisition stage: The top camera shoots continuously at 15fps The beam camera is triggered to capture the picture when the spreader moves above the container All images are transmitted to the edge terminal in real time via Gigabit Ethernet (2) Target detection stage: Use the improved YOLOv8s model for multi-target detection: Detection targets include: Roof number area (rotating rectangle) Marking area on each side of the container (polygonal box) Dangerous area (rectangular frame) Detection accuracy: mAP@0.5 reaches 98.7%; (3) OCR recognition stage: Roof number recognition: Using the PLRNet model, it supports tilt angle (±45°) recognition with a recognition accuracy of ≥99.2%.

[0066] Here is another example, which is a schematic illustration formed by combining the above examples.

[0067] Taking the rail crane recognition scenario as an example, the system is equipped with six 5-megapixel global shutter cameras, evenly mounted on the crane bracket, with a spacing of 1.2 meters. Each camera is equipped with a laser rangefinder for dynamic focus adjustment. To address vibration during lifting operations, rubber shock-absorbing pads are added to the camera mounts, and motion blur compensation technology based on Wiener filtering is incorporated into the image processing algorithm.

[0068] This application uses an independently developed embedded edge recognition terminal, which is convenient for deployment at the identification site of the port scene, with low cost, small size and low power consumption.

[0069] The OCR recognition algorithm of this application not only supports image snapshot recognition, but also supports video stream recognition, that is, continuous recognition using the camera's video stream during the movement of the container or vehicle. In this way, the algorithm can automatically obtain better image frames in the video stream for continuous recognition, ensuring higher recognition rate and accuracy.

[0070] This embodiment provides an electronic device, which can be expressed in the form of a computing device (for example, a server device), including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the embedded port edge identification method provided in the above embodiment can be implemented.

[0071] Figure 4 The hardware structure diagram of this embodiment is shown in FIG. Figure 4 As shown, the electronic device 30 specifically includes: At least one processor 31, at least one memory 32, and a bus 33 for connecting different system components (including the processor 31 and the memory 32), wherein: The bus 33 includes a data bus, an address bus, and a control bus.

[0072] The memory 32 includes a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0073] The memory 32 also includes a program / utility 325 having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0074] The processor 31 executes various functional applications and data processing by running the computer program stored in the memory 32, such as the steps of the embedded port edge identification method provided in the above embodiment of the present invention.

[0075] The electronic device 30 can further communicate with one or more external devices 34 (e.g., a keyboard, pointing device, etc.). This communication can occur via an input / output (I / O) interface 35. Furthermore, the electronic device 30 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. The network adapter 36 communicates with other modules of the electronic device 30 via a bus 33. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 30, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID (RAID) systems, tape drives, and data backup storage systems.

[0076] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the present application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0077] In this specification, the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments described later, the description is relatively simple, and the relevant parts can be referred to the partial description of the previous embodiments.

[0078] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An embedded port edge recognition method, characterized in that: Executed on embedded edge devices deployed at the port site, including: Receive a port operation video stream captured by an industrial camera, the video stream including: container trucks and containers; Convert the video stream into image frames and store them in a shared memory area; Based on an adaptive frame sampling algorithm, key frames are selected from the continuous image frames; Inputting the key frame into an improved target detection model to obtain target detection areas marked by rotating polygonal boxes and category labels corresponding to each target detection area; The improved target detection model is constructed in the following way: The backbone network is based on the YOLOv8 architecture; Add WIoU v3 loss function to dynamically focus on difficult samples; Integrate scale-compensated feature pyramids in the Neck layer to enhance multi-scale feature fusion; The output layer supports rotating polygons to adapt to oblique angles of the container. The categories of labels include: license plate, container number, danger label, seal and damage; Identify the target detection area. If the category label is a license plate, container number, or danger sign, call the license plate recognition model to perform character or pattern recognition on the target detection area. If the category label is a seal or damage, use the improved target detection model to classify and identify the target detection area. Output the recognition results and corresponding location coordinates to the port control system.

2. The embedded port edge recognition method according to claim 1, characterized in that: The screening of the image frames based on the adaptive frame sampling algorithm includes: Setting the first image frame of the video stream as an initial key frame and extracting feature points; For each subsequent image frame, the following checks are performed: Based on the optical flow method, the position of the feature points of the previous key frame in the current frame is tracked frame by frame. If the proportion of successfully tracked feature points is lower than the preset tracking rate threshold, the current frame processing is terminated; Based on the successfully tracked feature points, the optical flow vector mean and covariance matrix of all successfully tracked feature points are obtained, and if the trace of the covariance matrix is ​​greater than a preset motion stability threshold, the current frame processing is terminated; Count the proportion of optical flow abnormal points. If the proportion of abnormal points is greater than the preset occlusion tolerance threshold, terminate the current frame processing; Calculate the Laplace operator variance of the current frame. If the Laplace operator variance is less than the preset clarity threshold, terminate the current frame processing. If the current frame passes all judgments, the current frame is set as the new key frame and the feature points are updated.

3. The embedded port edge recognition method according to claim 1, characterized in that: The embedded port edge recognition method further includes: When a container truck enters the detection area, a photoelectric sensor is used to trigger an industrial camera to capture images of the truck and / or container to obtain key frames. The key frame is stored in the memory shared area after hard decoding.

4. The embedded port edge recognition method according to claim 1, characterized in that: The target detection model and the license plate recognition model are deployed on an NPU processor.

5. The embedded port edge recognition method according to claim 4, characterized in that: Convert the target detection model into an INT8 quantized RKNN format model; And / or, converting the license plate recognition model into an INT8 quantized RKNN format model.

6. The embedded port edge recognition method according to claim 1, characterized in that: Before identifying the target detection area, a C++ post-processing program accelerated by OpenMP is used to perform parallel processing on the target detection area, including: performing geometric correction and non-maximum suppression filtering on the rotated polygon box.

7. An embedded edge recognition terminal, characterized in that: For implementing the embedded port edge recognition method according to any one of claims 1 to 6, the embedded edge recognition terminal comprises: A communication interface unit comprising a Gigabit Ethernet interface and an RS485 interface; wherein the Gigabit Ethernet interface is used to receive a video stream of port operations captured by an industrial camera, the video stream including trucks and containers; and the RS485 interface is used to transmit the recognition results and corresponding location coordinates to the port control system; Processing unit, the processing unit adopts industrial-grade domestic processor chip, the industrial-grade domestic processor chip integrates: hard decoding module, NPU processor and CPU processor; The hardware decoding module is used to convert the video stream into image frames and store them in a shared memory area; the NPU processor is used to deploy an improved target detection model and license plate recognition model; and the CPU processor is used to run a Linux operating system. A customized circuit board, which is matched with the processing unit, includes a multi-layer PCB board; the customized circuit board is used to carry the communication interface unit, the processing unit and the storage unit memory.

8. The embedded edge recognition terminal according to claim 7, characterized in that: The embedded edge identification terminal further includes: a dual gigabit PHY chip and an isolation circuit; The dual Gigabit PHY chip is connected to the processing unit to form a redundant second Gigabit Ethernet interface to support automatic switching of optical fiber and copper cable; The isolated circuit is used to block electromagnetic interference in the port.

9. A port operation control system, characterized in that: The port operation control system includes: At least one industrial camera, a gigabit network switch, an embedded edge recognition terminal according to any one of claims 7 to 8, and a PLC controller; The industrial cameras are deployed at road crossings, rail cranes and / or bridge cranes to capture real-time video streams of trucks and containers. The gigabit network switch is used to connect the industrial camera to the embedded edge recognition terminal via gigabit Ethernet; The embedded edge recognition terminal is used to recognize the video stream and output the recognition result and the corresponding position coordinates; The PLC controller receives the recognition result and corresponding position coordinates of the embedded edge recognition terminal, and controls the gate or lifting equipment to perform corresponding operations according to the recognition result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the embedded port edge identification method according to any one of claims 1 to 6 is implemented.

Images