Container handling machine digital control auxiliary system and identification method

Abstract

The application discloses a container loading and unloading machine digital control auxiliary system and a recognition method, and relates to the technical field of container loading and unloading machine digital control auxiliary systems.

Description

Technical Field

[0001] This invention relates to the field of container loading and unloading and intelligent identification technology, and in particular to a digital control auxiliary system and identification method for container loading and unloading machinery. Background Technology

[0002] With the rapid development of China's railway logistics and transportation industry, intelligent and containerized freight transportation has become a trend, and the demand for refined management of large-scale freight yards is increasing. Currently, the manual issuance and reporting of work tasks via telephone, walkie-talkie, and paper documents, as well as the manual transcription of work data, are prone to problems such as inaccurate and falsified work data.

[0003] In existing technologies, containers, as the core unit of global logistics, require automatic identification and spatial positioning of their container numbers for automated port and yard operations. To achieve automated loading and unloading, equipment such as specialized loading and unloading vehicles using identification systems has been proposed. However, these technologies are still immature and suffer from numerous problems. For example, existing technologies lack robustness in container number identification. Current identification methods largely rely on single-frame image OCR, which is susceptible to variations in layout (horizontal / vertical, reverse, single / double line) and image quality (dirt, blur, reflection, occlusion), leading to missed detections and false detections. Furthermore, they lack the ability to correct random errors using temporal information. Spatial positioning is costly or has limited accuracy. Existing positioning technologies either rely on GPS / UWB solutions requiring sensor installation, which are costly; or on expensive equipment like LiDAR; or on monocular vision, which requires special markings or is susceptible to interference from corner features, making it difficult to meet the accuracy and robustness requirements of high-precision lifting. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a digital control auxiliary system for container loading and unloading machinery that enables automatic identification of container number / vehicle number and automatic collection of information such as container weight and container operation location.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] In a first aspect, this application discloses a digital control auxiliary system for container loading and unloading machinery, comprising: a container number recognition unit, including at least one camera, for acquiring images of the container surface during loading and unloading operations; a positioning data acquisition unit, including a fixedly installed and calibrated binocular vision camera, the binocular vision camera being used to simultaneously acquire left and right eye images of the surface where the container number area is located; and an on-board intelligent control host processing unit, communicatively connected to the container number recognition unit and the positioning data acquisition unit, the on-board intelligent control host processing unit being configured to: acquire the container number text content output by the container number recognition unit for the container; acquire the left and right eye images acquired by the binocular vision camera; based on the container number text content, determine a corresponding container number area matching the container number text content in the left and right eye images; perform visual feature extraction and stereo matching within the corresponding container number area determined in the left and right eye images, and reconstruct the three-dimensional spatial pose of the container number area according to the matching result to generate positioning data of the container.

[0007] Furthermore, the container number identification unit includes: a first camera, installed on the crane grab or crane beam of the loading and unloading machinery, for capturing multiple frames of continuous images of the container body; a second camera, installed on the loading and unloading machinery, for capturing multiple frames of continuous images of the train body to obtain the train number; and a supplementary lighting device for supplementing the shooting area of ​​the first camera and / or the second camera; wherein, the vehicle-mounted intelligent control host processing unit is also connected to the first camera and the second camera via Ethernet to receive the image sequences captured by them.

[0008] Furthermore, the positioning data acquisition unit also includes: a positioning module, comprising an RTK base station and a mobile station, wherein the mobile station is deployed on top of the loading and unloading machinery, and the RTK base station is deployed on a fixed building in the work area, for acquiring the latitude and longitude coordinates of the loading and unloading machinery; an altitude measurement module, comprising a UWB ranging base station and a ranging receiving station, wherein the ranging base station is deployed on top of the loading and unloading machinery, and the ranging receiving station is installed on the crane grab, for acquiring the height of the crane grab by calculating the UWB signal propagation time; the vehicle-mounted intelligent control host processing unit is connected to the positioning module and the altitude measurement module via a network port to receive the latitude and longitude coordinates and height data collected by them, as an auxiliary reference for the container positioning data.

[0009] Secondly, this application also discloses a container identification method for the aforementioned digital control auxiliary system for container loading and unloading machinery, comprising the following steps: S101, acquiring a sequence of multiple consecutive frames of images of the same container collected by the container number identification unit, and preprocessing each frame of the image; S102, performing layout method category detection and container number region position detection on each frame of the image to obtain the layout method and container number region bounding box corresponding to each frame of the image sequence; S103, performing geometric correction on the container number region of each frame of the image according to the detected layout method, uniformly converting it into a forward horizontal layout, and cropping out the corrected region image; S1 04. Perform character detection on the corrected region images of each frame to obtain a character detection set for each frame. The character detection set includes the bounding box coordinates, category, and confidence score of each character. S105. Transform the character detection coordinates of each frame to the same reference coordinate system, perform spatiotemporal clustering on the center points of all character detection boxes, and perform category voting and confidence score fusion for each cluster to generate a fused character set. S106. Sort and perform fault-tolerant completion on the fused character set according to the layout method to generate a main candidate sequence that conforms to the standard encoding rules. S107. Perform multi-frame collaborative verification on the main candidate sequence and output the finally confirmed box number text content.

[0010] Furthermore, the spatiotemporal clustering performed by the vehicle-mounted intelligent control host processing unit in step S105 includes: uniformly transforming the center coordinates of the character detection boxes in each frame to the coordinate system of the first frame correction region in the image sequence; removing image frames whose layout category or box number region position is inconsistent with other frames; and using a density-based clustering algorithm to cluster the center points of all valid character detection boxes to form multiple character clusters, including: clustering the centers of all character detection boxes in all frames, and processing each cluster... Calculate the average position Voting is conducted among the categories, and the category with the highest frequency is selected as the merged category. When tying votes, select the vote with the highest confidence level; calculate the fusion confidence level. ; Obtain the merged character set .

[0011] Furthermore, the fault-tolerant completion performed by the vehicle-mounted intelligent control host processing unit in step S106 includes: determining the character sorting rules according to the layout method and sorting the fused character set; calculating the spacing between adjacent characters after sorting and estimating the baseline character spacing using a robust statistical method; in response to a situation where the spacing between adjacent characters is significantly greater than the baseline character spacing, determining that there are missing characters and inserting a placeholder at the corresponding position; and inferring and completing any missing characters at the beginning or end by combining the standard length of the container number and the boundary box information of the container number area.

[0012] Furthermore, the multi-frame collaborative verification performed by the vehicle-mounted intelligent control host processing unit in step S107 includes: determining whether the main candidate sequence is complete and passes the check code verification; if it is complete and passes the verification, the main candidate sequence is directly output as the final box number text content; otherwise, the candidate sequences generated independently from each frame image are used to vote and fill the placeholders or low-confidence character positions in the main candidate sequence to generate a supplemented sequence and perform verification; if it still fails the verification, a backtracking search is performed in the limited candidate space until a sequence that passes the verification is found or the sequence with the highest confidence is output.

[0013] Furthermore, the method also includes the on-board intelligent control host processing unit performing the following container 3D positioning stage, including the following steps: S201, acquiring the left and right eye images synchronously acquired by the binocular vision camera, and the container number text content confirmed and output in step S107; S202, based on the container number text content, determining the corresponding container number region with content matching in the left and right eye images respectively; S203, extracting character visual feature points and their descriptors and pixel coordinates within the corresponding container number regions in the left and right eye images. And perform stereo matching to establish the correspondence between feature point pairs of the left and right eyes; S204, according to the calibration parameters of the binocular vision camera and the correspondence between the feature point pairs, calculate the three-dimensional coordinates of each matched feature point in the camera coordinate system, and perform planar fitting on the obtained three-dimensional point cloud to obtain the normal vector and spatial equation of the container number plane; S205, based on the normal vector of the container number plane, the three-dimensional coordinates of the center point of the container number, and the relationship between the standard geometric dimensions of the container and the fixed position of the container number on the surface of the container, solve the six-degree-of-freedom pose of the container relative to the binocular vision camera.

[0014] Furthermore, in step S202, the vehicle-mounted intelligent control host processing unit determines the corresponding box number region for content matching by: using optical character recognition technology to perform text detection and recognition on the left and right images respectively, and extracting character features to output the box number text content for associating the left and right image regions and the box number region image, as well as the visual feature points, descriptors, and pixel coordinates of each character in them. .

[0015] Furthermore, step S205 includes: given the standard dimensions of the container and the fixed position of the container number on the container surface, the normal vector n of the container number plane defines the orientation of one side of the container, combined with the three-dimensional coordinates of the center point of the container number. And the fixed geometric relationship between the container number and the container body allows for the calculation of the container's local coordinate system. Relative to the left camera coordinate system rotation Peaceful relocation This allows us to obtain the three-dimensional pose of the center point of the container's bottom surface, including: in, The container number center is in the container's local coordinate system. The coordinates in the diagram.

[0016] The beneficial effects of this invention are:

[0017] The vehicle-mounted intelligent control host processing unit is equipped with a container recognition method. This method can achieve stable and accurate recognition of container numbers in complex scenarios through multi-frame fusion and adaptive layout analysis. Utilizing the recognized container number region and character features, combined with binocular vision, it achieves high-precision 3D pose calculation of the container without additional markings. The recognition and positioning processes are integrated in a closed loop, using recognized content to assist visual matching and spatial consistency to verify the recognition results, thereby improving the overall system performance. Attached Figure Description

[0018] Figure 1 This is a simplified schematic diagram of a digital control auxiliary system for container loading and unloading machinery according to some embodiments of this application;

[0019] Figure 2 This is a simplified schematic diagram of the overall algorithm flow of the digital control and identification method for container loading and unloading machinery according to some embodiments of this application;

[0020] Figure 3 This is a modular schematic diagram of a digital control and identification method for container handling machinery according to some embodiments of this application;

[0021] Figure 4 This is a simplified flowchart illustrating the multi-frame fusion process of a digital control and identification method for container handling machinery according to some embodiments of this application.

[0022] Figure 5 This is a simplified schematic diagram of binocular matching and three-dimensional reconstruction of a digital control and identification method for container handling machinery according to some embodiments of this application. Detailed Implementation

[0023] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] First, let's explain some technical terms:

[0025] An RTK base station is a high-precision satellite positioning auxiliary device that receives GNSS (such as BeiDou and GPS) signals, calculates local errors, and sends differential correction data to the mobile station in real time.

[0026] A UWB ranging base station is a base station that performs high-precision two-way ranging with UWB tags or receives tag signals to calculate location. A UWB ranging link refers to a wireless communication channel established between UWB devices (such as a tag and a base station, or two tags) for accurate distance measurement.

[0027] F-TR locks are container locking devices (turnlocks) used on railway flatcars to securely fasten containers to the railway flatcars and prevent them from shifting or falling during transport.

[0028] according to Figures 1-5 This application describes a digital control auxiliary system and identification method for container loading and unloading machinery according to embodiments of the present application.

[0029] like Figure 1 A digital control auxiliary system for container loading and unloading machinery according to an embodiment of this application includes a container number recognition unit, including at least one camera for acquiring images of the container surface during loading and unloading operations; a positioning data acquisition unit, including a fixedly installed and calibrated binocular vision camera, the binocular vision camera being used to simultaneously acquire left and right eye images of the surface where the container number area is located; and an on-board intelligent control host processing unit, which is communicatively connected to the container number recognition unit and the positioning data acquisition unit, respectively. The on-board intelligent control host processing unit is configured to: acquire the container number text content output by the container number recognition unit for the container; acquire the left and right eye images acquired by the binocular vision camera; determine the corresponding container number area matching the container number text content in the left and right eye images based on the container number text content; perform visual feature extraction and stereo matching within the corresponding container number area determined in the left and right eye images, and reconstruct the three-dimensional spatial pose of the container number area according to the matching result to generate the positioning data of the container.

[0030] The container number identification unit is a device that can obtain a globally unified and clearly printed container number from the surface of the container based on image acquisition and intelligent recognition methods. This identification is used as an accurate text identifier. The binocular vision camera in the positioning data acquisition unit can include a vision sensor consisting of two cameras with fixed relative positions and calibrated internal and external parameters. It can simultaneously acquire two images of the same scene for stereo vision calculation.

[0031] The aforementioned vehicle-mounted intelligent control host processing unit is the core of the entire system's control and processing. It is responsible for the centralized control and processing of the container number recognition unit, vehicle number recognition unit, PLC data acquisition unit, positioning data acquisition unit, and empty / loaded mixed loading early warning unit. It also uploads the acquired and recognized operational data to the vehicle-mounted terminal via a serial port. The system in this embodiment utilizes the globally unified and clearly printed container number identifiers inherent in the containers, using them as accurate text identifiers and as a stable and rich source of visual spatial features. This enables the coordinated use of container number text information and visual features to automate the entire process from accurate identification to precise positioning.

[0032] In some embodiments, the digital control auxiliary system for container loading and unloading machinery also includes a vehicle number recognition unit, a PLC data acquisition unit, an on-board terminal, an empty / loaded mixed loading early warning unit, and an expansion unit.

[0033] As mentioned above, the vehicle-mounted intelligent control host processing unit is responsible for the centralized control and processing of the container number recognition unit, vehicle number recognition unit, PLC data acquisition unit, positioning data acquisition unit, and empty / heavy mixed loading early warning unit, and uploads the collected and recognized operation data to the vehicle-mounted operation terminal via a serial port. The container number capture and recognition unit is installed on the crane grab and crane crossbeam, and takes pictures through a camera and is equipped with supplementary lighting equipment. It is connected to the vehicle-mounted intelligent control host processing unit via an RJ45 fiber optic cable. The train number capture and recognition unit is installed on the crane (the installation position varies depending on the type of crane). The PLC data acquisition unit is interconnected with the crane's electronic control system, and collects container weight information and spreader lock status information, all of which are connected to the vehicle-mounted intelligent control host processing unit via an RJ45 serial port.

[0034] The positioning data acquisition unit is divided into a positioning section and an altitude measurement section. The positioning section includes an RTK base station and a mobile station. The mobile station is deployed on the top of the crane, and the RTK base station is deployed on the top of a nearby building. The altitude measurement section includes a ranging base station and a ranging receiving station. The vehicle-mounted intelligent control host processing unit and the vehicle-mounted terminal are installed inside the loader's cab. The empty / loaded mixed loading early warning unit is deployed inside the vehicle-mounted intelligent control host processing unit.

[0035] In this embodiment, image intelligent recognition technology is used to automatically collect train car numbers and container numbers; a PLC acquisition unit is used to automatically collect container weight and lock status; RTK is used to automatically collect container position and height; a rule algorithm is used to determine whether two container numbers belong to the same train, thereby performing empty-load mixed loading detection and prediction; these data are transmitted to the intelligent host via Ethernet / serial port, etc.; the intelligent host sends data to the on-board terminal via serial port to achieve automated collection of crane / forklift container operation data. This provides support for improving the efficiency and safety management level of the terminal. Based on advanced image intelligent recognition technology, images of the train car body and container body are captured, container numbers are identified, and train car numbers are identified, assisting container loading and unloading machinery in achieving automated collection and intelligent management of operation data during operation. Image recognition technology, mobile communication technology, serial data communication technology, RTK positioning technology, and other technologies are used to achieve automatic collection and digital transmission of operation information.

[0036] The digital control auxiliary system for container loading and unloading machinery is powered on-board through the machinery. Once the machinery is powered on, the user activates the intelligent control host, and the system automatically starts and connects to relevant equipment (cameras, PLC data acquisition and loading, on-board terminals, etc.). The user can view the work tasks assigned to them by the freight handler through the on-board terminal. During the operation of the loading and unloading machinery, the container number capture and recognition unit and the train number capture and recognition unit automatically capture and monitor the train car and container bodies in real time. The on-board intelligent control host processing unit performs content recognition on the captured images, automatically identifying the train car number and container number in the images. Simultaneously, the on-board intelligent control host processing unit verifies the identified results according to rules such as container number coding rules and train number coding rules. For railway containers, the system can automatically complete the identification results for missing / incorrect container owner codes and cache the verified and completed train car number and container number for later processing. The PLC data acquisition unit continuously collects container weight information and spreader status information and transmits it to the intelligent host via Ethernet. The positioning data acquisition unit continuously collects the container's location and height information and transmits the data to the intelligent host via Ethernet. The intelligent host then transmits the container number, vehicle number, spreader status, container weight, container location information, and container layer number to the vehicle-mounted terminal via serial port.

[0037] Specifically, the container number identification unit includes a first camera and a supplementary lighting device. The first camera is installed on the crane gripper or crane beam of the loading and unloading machinery to capture multiple frames of continuous images of the container body. The second camera is installed on the loading and unloading machinery to capture multiple frames of continuous images of the train car body to obtain the train number. The supplementary lighting device is used to provide supplementary lighting to the shooting areas of the first camera and / or the second camera. The vehicle-mounted intelligent control host processing unit is connected to the first and second cameras via Ethernet to receive the captured image sequences. The first camera is preferably a high-definition camera, and the second camera is a regular camera. The container number identification unit is connected to the vehicle-mounted intelligent control host processing unit via Ethernet. The PLC data acquisition unit interfaces with the crane PLC electronic control system via hardwiring or an industrial bus to collect spreader lock status information and container weight information, and connects to the vehicle-mounted intelligent control host processing unit via a network port.

[0038] The positioning data acquisition unit further includes a positioning module and an altitude measurement module. The positioning module includes an RTK base station and a mobile station. The mobile station is deployed on top of the loading and unloading machinery, and the RTK base station is deployed on a fixed building in the work area to obtain the latitude and longitude coordinates of the loading and unloading machinery. The altitude measurement module includes a UWB ranging base station and a ranging receiving station. The ranging base station is deployed on top of the loading and unloading machinery, and the ranging receiving station is installed on the crane grab to obtain the height of the crane grab by calculating the UWB signal propagation time. Specifically, the RTK base station continuously receives satellite positioning signals and sends differential correction data. The rover simultaneously receives satellite signals and differential data from the base station. After real-time differential calculation to eliminate positioning errors and complete coordinate conversion, it outputs the precise latitude and longitude of the gantry crane gripper. The ranging base station for the height measurement section is deployed near the top of the crane, and the ranging receiving station is installed on the gantry crane gripper. The two form a UWB ranging link. The ranging base station sends UWB pulse signals, and the receiving station calculates the straight-line distance by calculating the signal propagation time and combining it with the UWB signal propagation speed. Then, based on the known installation height and relative position of the ranging base station, it performs geometric conversion to obtain the actual height of the gripper, achieving accurate measurement of the latitude, longitude, and height of the gantry crane gripper. The onboard intelligent control host processing unit is connected to the positioning module and the height measurement module via a network port to receive the collected latitude, longitude, and height data as an auxiliary reference for the container positioning data.

[0039] The aforementioned empty-load mixed loading early warning unit calculates whether adjacent containers are on the same train as the currently operated container based on the container's landing location information. If two containers are on the same train, it calculates whether there is an empty-load mixed loading situation between them. The on-board intelligent control host processing unit is the core of the entire system's control and processing, responsible for the centralized control and processing of the container number identification unit, vehicle number identification unit, PLC data acquisition unit, positioning data acquisition unit, and empty-load mixed loading early warning unit. It also uploads the collected and identified operation data to the on-board terminal via a serial port. The on-board terminal is connected to the on-board intelligent control host processing unit via a serial port.

[0040] In some embodiments, the system further includes expansion units, such as: a container overweight / over-deviation acquisition unit, an F-TR anti-snagging device signal acquisition unit, a limit switch signal acquisition unit, a crane anti-collision device signal acquisition unit, and a vehicle license plate acquisition and recognition unit. These expansion units integrate mature systems and products, and perform information integration and interaction through the onboard intelligent control host processing unit.

[0041] The system boasts wider adaptability, effectively recognizing characters in images with distortion, low light, dirt, or smoke. Addressing the issue of inaccurate train and container number recognition in existing technologies, this invention utilizes advanced intelligent image recognition technology. Through continuous image capture and content recognition of container and train bodies, and comparison of multiple image recognition results from various cameras, it effectively improves the accuracy of container and train number recognition. Furthermore, by automatically completing the container number recognition results using container number coding rules, it further enhances the accuracy of container number recognition. The system automatically collects and transmits operational data (including train number, container number, container weight, container location, and empty / load detection warnings) during container handling, replacing the manual copying, paper transmission, and post-processing of data. This reduces manual intervention, lowers workload, avoids errors caused by human factors, and improves operational efficiency and accuracy. The system provides an information management tool for container handling operations in railway freight yards, logistics parks, distribution stations, and dedicated lines. It provides digital support and assurance for the scheduling of container handling operations for customers such as railway freight yards, logistics parks, distribution stations, and dedicated lines.

[0042] Based on the container loading and unloading machinery digital control auxiliary system described in the foregoing embodiments, the on-board intelligent control host processing unit is equipped with a container recognition method. This method can achieve stable and accurate recognition of container numbers in complex scenarios through multi-frame fusion and adaptive layout analysis. Utilizing the recognized container number area and character features, combined with binocular vision, high-precision three-dimensional pose calculation of the container is achieved without additional markings. The recognition and positioning processes are integrated in a closed loop, using the recognized content to assist visual matching and spatial consistency to verify the recognition results, thereby improving the overall system performance.

[0043] Combination Figure 2 and Figure 3 According to the container identification method of this application embodiment, which is mounted on the vehicle-mounted intelligent control host processing unit, the container identification method includes the following steps:

[0044] First, combined Figure 4 Understand the process of multi-frame fusion.

[0045] S101. Obtain a sequence of multiple consecutive frames of images of the same container collected by the container number recognition unit. Each frame of the image is preprocessed, including illumination normalization and noise reduction.

[0046] S102. Perform layout method category detection and box number region location detection on each frame of the image to obtain the layout method and box number region bounding box corresponding to each frame of the image sequence. The layout method category includes forward horizontal single line, horizontal double line, forward vertical line, reverse layout, etc. The layout method category and box number region bounding box can be detected simultaneously using an object detection model (such as YOLO series or other object detection models).

[0047] S103. Perform geometric correction on the box number area of ​​each frame image according to the detected layout method, including rotation, flipping and other operations, and convert it into a positive horizontal layout, and crop out the corrected area image.

[0048] S104. Perform character detection on the corrected region images of each frame to obtain the character detection set for each frame:

[0049]

[0050] in, The coordinates of the center of the character box For categories (0-9, AZ), Here, m represents the confidence level, and m is the frame index, corresponding to the m-th frame image. is the total number of characters detected in the m-th frame (i.e., the number of character boxes in this frame), i is the character index detected in the i-th frame, and in this example, the value range is (1,2…M), and M is the total number of frames.

[0051] S105. Transform the character detection coordinates of each frame to the same reference coordinate system, perform spatiotemporal clustering on the center points of all character detection boxes, and perform category voting and confidence fusion for each cluster to generate a fused character set; specifically, uniformly transform the center coordinates of the character detection boxes of each frame to the coordinate system of the first frame correction area in the image sequence, and then remove frames with inconsistent layout or regions, so as to cluster the centers of all character detection boxes in all frames (such as DBSCAN). In this embodiment, a density-based clustering algorithm is used to cluster the center points of all valid character detection boxes to form multiple character clusters; cluster the centers of all character detection boxes in all frames, and perform category voting and confidence fusion on each cluster. :

[0052] Calculate the average position .

[0053] The categories were voted on, and the category with the highest frequency was selected as the merged category. In the event of a tie, the vote with the highest confidence level is selected.

[0054] Calculate fusion confidence ; Obtain the merged character set .

[0055] S106. The fused character set is sorted and error-tolerantly completed according to the layout method to generate a main candidate sequence conforming to standard encoding rules. Specifically, the error-tolerant completion performed by the vehicle-mounted intelligent control host processing unit in step S106 includes: determining the character sorting rules according to the layout method, sorting the fused character set, taking a single horizontal row as an example, and arranging it in ascending order by the horizontal axis; calculating the spacing between adjacent characters after sorting, and estimating the baseline character spacing using robust statistical methods such as the median; in response to a spacing between adjacent characters that is significantly greater than the baseline character spacing, for example, the spacing is greater than a specified multiple of the baseline spacing, it is determined that there is a missed character, and a placeholder is inserted at the corresponding position; combining the standard length of the container number, such as 11 digits, and the boundary box information of the container number area, it is inferred and completed the characters that may be missing at the beginning or end. For other layout methods such as double-line, vertical, and reverse layout, corresponding strategies are used to process them, and finally a main candidate sequence that may contain placeholders is generated.

[0056] For example, taking a single row in the forward horizontal direction as an example, the process is as follows:

[0057] Sort by Sort in ascending order.

[0058] Baseline spacing estimation, calculating the spacing between adjacent characters. The baseline spacing is obtained through robust estimation using the median and other methods. .

[0059] Missed detection inference, if the spacing between adjacent characters Significantly larger than the reference spacing ,like > If the result is not found, a placeholder is inserted.

[0060] The beginning and end are completed by combining the standard length (11 bits) and the region boundary to infer the beginning and end missing parts.

[0061] In this example, other layout methods (such as double-line, vertical, and reverse) are handled with corresponding strategies, ultimately generating a primary candidate sequence of length 11 that may contain placeholders. .

[0062] S107. Perform multi-frame collaborative verification on the main candidate sequence and output the finally confirmed box number text content.

[0063] Specifically, the multi-frame collaborative verification performed by the vehicle intelligent control host processing unit in step S107 includes: determining the main candidate sequence. Check if the sequence is complete and passes ISO 6346 verification; if complete and passes verification, output the main candidate sequence as the final box number text content; otherwise, use the candidate sequences generated independently from each frame image. For the main candidate sequence The placeholders or low-confidence characters in the sequence are filled by voting to generate the supplemented sequence. Then perform a verification; if it still fails the verification, backtrack the search in the limited candidate space until a sequence that passes the verification is found or the sequence with the highest confidence is output.

[0064] In some embodiments, the container identification method further includes a container three-dimensional positioning stage performed by the on-board intelligent control host processing unit, referring to... Figure 5 Understanding specifically includes the following steps:

[0065] S201. Obtain the left and right eye images synchronously acquired by the binocular vision cameras, and the box number text content confirmed and output in step S107; specifically, based on a fixedly installed and calibrated pair of cameras, obtain the intrinsic parameter matrix. And the extrinsic parameters of the right camera relative to the left camera, including the rotation matrix R, translation vector T, and baseline. .

[0066] S202. Based on the container number text content, determine the corresponding container number region with matching content in the left eye image and the right eye image respectively; In this embodiment, the vehicle intelligent control host processing unit uses optical character recognition technology to perform text detection and recognition on the left eye image and the right eye image respectively, and matches the recognized text content with the container number text content confirmed and output in step S107.

[0067] Specifically, optical character recognition technology is used to perform text detection and recognition on the left and right images respectively, and character feature extraction is performed to output the box number text content for associating the left and right image regions and the box number region image with the visual feature points, descriptors, and pixel coordinates of each character in them. , Let be the pixel coordinates of the i-th feature point in the left image. The pixel coordinates of the feature point in the right image that matches the i-th feature point in the left image.

[0068] 203. Within the corresponding box number regions of the left and right eye images, extract character visual feature points, their descriptors, and pixel coordinates. The visual feature points can be corner points, centroids, etc. Perform stereo matching, and combine the feature descriptors and prior spatial distribution of box number characters to establish the correspondence between feature point pairs in the left and right eye images. N represents the total number of feature point pairs that have successfully established stereo matching, i.e., the number of feature points that have been successfully matched in the left and right images.

[0069] S204. Based on the correspondence between the calibration parameters of the binocular vision camera and the feature point pairs, calculate the three-dimensional coordinates of each matched feature point in the camera coordinate system, and perform plane fitting on the obtained three-dimensional point cloud to obtain the normal vector and spatial equation of the box number plane.

[0070] Specifically, the calibration of the stereo camera includes obtaining the intrinsic parameter matrix and the extrinsic parameters (rotation matrix, translation vector, baseline) of the right camera relative to the left camera. For each pair of matched feature points, its three-dimensional coordinates are solved based on the camera model. (In the left camera coordinate system). For a calibrated ideal system, depth Calculated from parallax, focal length, and baseline:

[0071]

[0072] in, Let f be the parallax and f be the focal length. , These are the x-coordinates of the corresponding feature points in the left and right images, respectively (in this example, it is assumed that the rows are aligned after epipolar correction).

[0073] In some embodiments, it can also be obtained by solving the projection equation.

[0074] For the obtained 3D point cloud set The box-number plane equations were fitted using principal component analysis (PCA) or RANSAC. Where n is the unit normal vector, all points P satisfying this equation lie on the plane, where P is the 3D point coordinate vector, and the corresponding set is the 3D point cloud set. .

[0075] S205. Based on the plane normal vector of the container number, the three-dimensional coordinates of the center point of the container number, and the relationship between the standard geometric dimensions of the container and the fixed position of the container number on the surface of the container, calculate the six-degree-of-freedom pose of the container relative to the binocular vision camera.

[0076] Specifically, given the standard dimensions of the container (length L, width W, height H) and the fixed position of the container number on the container surface (e.g., the offset of the center point from a corner point), In the container local coordinate system (Below), the normal vector n of the container number plane defines the orientation of one side of the container, combined with the three-dimensional coordinates of the center point of the container number. (Obtained from the center of the fitted planar point cloud or the centroid of feature points) and the fixed geometric relationship between the container number and the container body, the local coordinate system of the container can be solved. Relative to the left camera coordinate system rotation Peaceful relocation This allows us to obtain the three-dimensional pose of key points such as the center of the container's bottom surface, including:

[0077]

[0078] in, It is the center of the container number. The coordinates in the equation are given. Combining the alignment constraint between n and the normal to the side of the container, the rotation is obtained. Peaceful relocation This allows for further calculation of the three-dimensional pose of key points such as the center of the container's bottom surface.

[0079] In some embodiments, S205 further includes introducing direction information, in this example using the horizontal direction of the box number character as a third constraint.

[0080] Specifically, the complete rotation is determined using the character orientation; for example, since each character frame has been detected in S202, two orthogonal directions (horizontal and vertical) of the container number plane can be extracted by the arrangement of the character frame centers or the main direction of the region. These directions are in the container's local coordinate system. The value is known. The specific steps and calculations are as follows:

[0081] First, extract the orthogonal direction (camera coordinate system) of the box number plane.

[0082] The center points of all character boxes obtained from S104 (Image coordinates), obtained by least-squares fitting of a straight line or PCA, include two principal directions:

[0083] Horizontal direction , which indicates the direction of character arrangement (usually along the length of the container).

[0084] vertical direction , perpendicular to the horizontal and located in the plane of the box number, by We obtain (n is the unit normal vector).

[0085] Since the directions in the image are two-dimensional, they need to be back-projected onto a three-dimensional plane. In this example, the two-dimensional directions are mapped to three-dimensional spatial vectors (i.e., vectors on the plane) using the fitted box-number plane equation. A simpler method is to directly use the three-dimensional point cloud fitted in S204 and obtain the principal directions by projecting the point cloud onto the two-dimensional distribution on the plane.

[0086] Finally, we obtain a pair of orthogonal unit vectors in the camera coordinate system, including the horizontal direction. Vertical direction .

[0087] Specifically, a local coordinate system for the container can be defined. The corresponding directions include:

[0088] Let's take the case where the container number is affixed to the right side of the container as an example:

[0089] The side normal vector is China (Pointing outwards in the width direction).

[0090] The horizontal vector of the character along the length of the container is .

[0091] The vertical vector of the character along the height direction is .

[0092] If the box number is on another side, simply replace the corresponding one. That's all.

[0093] Then, solve for the rotation matrix.

[0094] Based on the correspondence between two orthogonal directions:

[0095] The third axis is obtained by the cross product:

[0096] Note that the direction signs must be consistent (if extracted). (The sign can be negative if the direction is opposite to the desired direction). The rotation matrix is:

[0097]

[0098] because The basis of is the orthogonal identity matrix, and the inverse of the right-hand matrix is ​​equal to its transpose, therefore it is also equal to:

[0099]

[0100] Then, solve for the translation vector.

[0101] Corresponding to the center point: ,in, For the center of the container number The coordinates in the image are as described above. .

[0102] have to:

[0103] .

[0104] Finally, the 3D pose of the container's key points is calculated, including:

[0105] definition The origin is the bottom left corner of the container's bottom surface (i.e., the corner point), and the coordinate axes are: Along the length, Y is along the width, and Z is vertically upward. For example, the key point is... The coordinates in the diagram include: the center of the bottom surface (L / 2, W / 2, 0); the center of the top surface (L / 2, W / 2, H); and the four corners of the bottom surface are ((0,0,0),(L,0,0),(0,W,0),(L,W,0)).

[0106] The formula for transforming each keypoint to the left camera coordinate system C1 is:

[0107]

[0108] in, Let the center of the bottom surface be the center of the plane. Continuing with the example above, the coordinates of the center of the bottom surface in the camera coordinate system are:

[0109]

[0110] In some implementations, if the extracted and If they are not completely orthogonal, they need to be orthogonalized first. The direction sign can be determined by the dot product sign: if... If the direction is opposite to the desired direction, then the vector is negative. Furthermore, all vectors must be normalized.

[0111] Through the above process, the 6-DOF pose of the container can be uniquely determined, and the 3D coordinates of all key points of interest can be output for subsequent grasping, positioning, or path planning.

[0112] Therefore, the above method first obtains the container number text and character features through binocular image input in the first stage, which greatly constrains and simplifies the binocular matching problem in the text localization process. Then, the spatial pose calculated in the second stage can be fed back to verify the spatial rationality of the recognition result (e.g., characters should be approximately coplanar). Finally, the accurate container number and its six-degree-of-freedom pose in three-dimensional space are output. Thus, multi-frame information can be fused to resist instantaneous interference, adaptive typesetting can handle diversity, and closed-loop verification can ensure the correctness of the result, thereby achieving a comprehensive improvement in recognition robustness. It directly utilizes the inherent container number features without modifying the container; binocular vision provides accurate three-dimensional information; text content assists in matching, improving reliability and accuracy, thereby achieving high positioning accuracy and low cost; recognition provides strong semantic association for positioning, and positioning provides spatial consistency verification for recognition. The fusion of the two forms an enhanced closed loop from perception to understanding to positioning, demonstrating the advantages of collaborative processing between the two stages. Furthermore, the method is based on an integrated solution and can be deployed in scenarios such as gates, quay cranes, and yard cranes, simultaneously meeting the automation requirements of identity recognition and accurate positioning.

[0113] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A digital control auxiliary system for container loading and unloading machinery, characterized in that, include: The container number identification unit includes at least one camera for capturing images of the container surface during loading and unloading operations; The positioning data acquisition unit includes a fixedly installed and calibrated binocular vision camera, which is used to simultaneously acquire left and right eye images of the surface where the container number area is located; The vehicle-mounted intelligent control host processing unit is communicatively connected to both the box number identification unit and the positioning data acquisition unit. The vehicle-mounted intelligent control host processing unit is configured as follows: Obtain the container number text content output by the container number recognition unit for the container; Acquire the left and right eye images captured by the binocular vision camera; Based on the container number text content, determine the corresponding container number region that matches the container number text content in the left eye image and the right eye image; Visual features are extracted and stereo matching is performed within the corresponding container number area determined in the left and right eye images. The three-dimensional spatial pose of the container number area is reconstructed based on the matching results to generate the positioning data of the container.

2. The digital control auxiliary system for container loading and unloading machinery according to claim 1, characterized in that, The box number identification unit includes: The first camera is installed on the crane grab or crane beam of the loading and unloading machinery to capture multiple frames of the container body continuously. The second camera, installed on the loading and unloading machinery, is used to capture multiple frames of continuous images of the train car body to obtain the train car number. A supplementary lighting device is used to provide supplementary lighting to the shooting area of ​​the first camera and / or the second camera; The vehicle-mounted intelligent control host processing unit is also connected to the first camera and the second camera via Ethernet to receive the image sequences captured by them.

3. The digital control auxiliary system for container loading and unloading machinery according to claim 1, characterized in that, The positioning data acquisition unit also includes: The positioning module includes an RTK base station and a mobile station. The mobile station is deployed on the top of the loading and unloading machinery, and the RTK base station is deployed on a fixed building in the work area to obtain the latitude and longitude coordinates of the loading and unloading machinery. The height measurement module includes a UWB ranging base station and a ranging receiving station. The ranging base station is deployed on the top of the loading and unloading machinery, and the ranging receiving station is installed on the crane grab. It is used to obtain the height of the crane grab by calculating the UWB signal propagation time. The vehicle-mounted intelligent control host processing unit is connected to the positioning module and the altimeter module via a network port to receive the latitude and longitude coordinates and altitude data collected by them, which serve as an auxiliary reference for the container positioning data.

4. A container identification method, used in a digital control auxiliary system for container loading and unloading machinery as described in any one of claims 1 to 3, characterized in that, Includes the following steps: S101. Obtain a sequence of multiple consecutive frames of images of the same container collected by the container number identification unit, and preprocess each frame of image. S102. Perform layout method category detection and box number area position detection on each frame of the image to obtain the layout method and box number area bounding box corresponding to each frame of the image sequence. S103. Based on the detected layout, perform geometric correction on the box number area of ​​each frame image, convert it into a uniform positive horizontal layout, and crop out the corrected area image. S104. Perform character detection on the corrected region images of each frame to obtain a character detection set for each frame. The character detection set includes the bounding box coordinates, category, and confidence level of each character. S105. Transform the character detection coordinates of each frame to the same reference coordinate system, perform spatiotemporal clustering on the center points of all character detection boxes, and perform category voting and confidence fusion for each cluster to generate a fused character set; S106. Sort and perform error-tolerant completion on the fused character set according to the layout method to generate a main candidate sequence that conforms to the standard encoding rules; S107. Perform multi-frame collaborative verification on the main candidate sequence and output the finally confirmed box number text content.

5. The container identification method according to claim 4, characterized in that, The spatiotemporal clustering performed by the vehicle intelligent control host processing unit in step S105 includes: The coordinates of the center of the character detection box in each frame are uniformly transformed to the coordinate system of the first frame correction area in the image sequence; Remove image frames whose layout type or box number area is inconsistent with other frames; A density-based clustering algorithm is used to cluster the center points of all valid character detection boxes, forming multiple character clusters, including: Cluster the centers of all character detection boxes in all frames, and for each cluster... : Calculate the average position ; The categories were voted on, and the category with the highest frequency was selected as the merged category. In the event of a tie, the vote with the higher confidence level is selected. Calculate fusion confidence ; Obtain the merged character set .

6. The container identification method according to claim 4, characterized in that, The fault-tolerant compensation performed by the vehicle intelligent control host processing unit in step S106 includes: Based on the layout method, the character sorting rules are determined, and the fused character set is sorted. Calculate the spacing between adjacent characters after sorting, and estimate the spacing between the baseline characters using a robust statistical method; In response to a situation where the spacing between adjacent characters is significantly greater than the baseline character spacing, it is determined that there is a missed character, and a placeholder is inserted at the corresponding position; By combining the standard length of the container number with the boundary information of the container number area, the missing characters at the beginning or end can be inferred and completed.

7. The container identification method according to claim 4, characterized in that... The multi-frame collaborative verification performed by the vehicle-mounted intelligent control host processing unit in step S107 includes: Determine whether the main candidate sequence is complete and has passed the checksum verification; If the sequence is complete and passes verification, the main candidate sequence will be output directly as the final box number text content. Otherwise, the candidate sequences generated independently from each frame image are used to vote and fill the placeholders or low-confidence character positions in the main candidate sequence, generating a supplementary sequence and verifying it. If the verification still fails, a backtracking search is performed within the limited candidate space until a sequence that passes the verification is found or the sequence with the highest confidence is output.

8. The container identification method according to any one of claims 1-7, characterized in that, The method further includes the on-board intelligent control host processing unit performing the following container three-dimensional positioning stage, including the following steps: S201. Obtain the left and right eye images synchronously acquired by the binocular vision camera, and the box number text content confirmed and output in step S107. S202. Based on the box number text content, determine the corresponding box number region with content matching in the left eye image and the right eye image respectively; S203. In the corresponding box number areas of the left and right eye images, extract the character visual feature points, their descriptors and pixel coordinates, and perform stereo matching to establish the correspondence between the feature point pairs of the left and right eye images. S204. Based on the correspondence between the calibration parameters of the binocular vision camera and the feature point pairs, calculate the three-dimensional coordinates of each matched feature point in the camera coordinate system, and perform plane fitting on the obtained three-dimensional point cloud to obtain the normal vector and spatial equation of the box number plane. S205. Based on the plane normal vector of the container number, the three-dimensional coordinates of the center point of the container number, and the relationship between the standard geometric dimensions of the container and the fixed position of the container number on the surface of the container, calculate the six-degree-of-freedom pose of the container relative to the binocular vision camera.

9. The container identification method according to claim 8, characterized in that, In step S202, the vehicle-mounted intelligent control host processing unit determines the corresponding box number area for content matching, including: Optical character recognition (OCR) technology is used to perform text detection and recognition on the left and right images, and character feature extraction is performed to output the box number text content. This text is used to associate the left and right image regions and the box number region image with the visual feature points, descriptors, and pixel coordinates of each character. .

10. The container identification method according to claim 8, characterized in that, Step S205 includes: Given the standard dimensions of the container and the fixed position of the container number on the container surface, the plane normal vector n of the container number defines the orientation of one side of the container, combined with the three-dimensional coordinates of the center point of the container number. And the fixed geometric relationship between the container number and the container body allows for the calculation of the container's local coordinate system. Relative to the left camera coordinate system rotation Peaceful relocation This allows us to obtain the three-dimensional pose of the center point of the container's bottom surface, including: in, The container number center is in the container's local coordinate system. The coordinates in the equation are used; combined with the alignment constraint between the plane normal vector n and the container side normal, the rotation is obtained by solving. Peaceful relocation ; The center points of all character boxes obtained from S104 The horizontal direction can be obtained by least squares fitting of a straight line or PCA. and vertical direction Then, using the 3D point cloud fitted in S204, the principal direction is obtained by projecting the point cloud onto the 2D distribution on the plane, thus obtaining a pair of orthogonal unit vectors in the camera coordinate system, including the horizontal direction. Vertical direction And define the side normal vector. Length direction vector Altitude direction vector ; The rotation matrix is: Translation vector for: The formula for transforming each keypoint to the left camera coordinate system C1 is: 。

Images