Port rmg or rtg double container automated operation system based on visual situation awareness

Abstract

The application discloses a port RMG or RTG double-container automatic operation system based on visual situation awareness, through setting a container number identification module, a layer height scanning module, a container seam gap detection module and an operation instruction conversion module, the front container and the rear container can be simultaneously identified, the identified container numbers and instruction information are matched, the double-container layer height scanning detection is carried out, the low-bee container height information and the high-bee container height information are stored, when it is found that the operation layer height information does not match the system field diagram information before operation, the operation is stopped to avoid safety risks, two sets of redundant schemes of a gap scanner and visual detection are arranged, the detection results are cross-checked, thereby realizing the basic alignment of the middle lock X direction, the double-container operation is realized through the TOS interface and the single-container operation production system, according to the matching rule, the 20ft single container meeting the double-container operation is converted into a double-container instruction, and the problem that the original production system has low support for the double-container operation task is solved.

Description

Technical Field

[0001] This invention relates to the field of port RMG or RTG automation implementation technology, and particularly to a port RMG or RTG dual-container automated operation system based on visual situational awareness. Background Technology

[0002] During the implementation or upgrade of port RMG or RTG automation, some terminals have adopted dual-container spreaders to improve efficiency. Dual-container spreaders can simultaneously handle two 20-foot containers, significantly increasing operational efficiency. However, dual-container operations are far more complex than single-container operations. During automation upgrades, the original single-container TOS production system cannot meet the demands of dual-container operations. Therefore, we propose a port RMG or RTG dual-container automated operation system based on visual situational awareness. Summary of the Invention

[0003] The main objective of this invention is to provide an automated port RMG or RTG dual-container operation system based on visual situational awareness, which can effectively solve the problems in the background art.

[0004] The technical solution adopted in this invention is as follows:

[0005] A port RMG or RTG dual-container automated operation system based on visual situational awareness, the operation system comprising:

[0006] The container number recognition module uses an AI artificial intelligence recognition algorithm to identify the container number on the top of the container and the container number on the door, and generates the container number recognition result.

[0007] The layer height scanning module is used to scan the adjacent shell boxes of the trolley, acquire and store the height information of the low shell box and the height information of the high shell box;

[0008] Box gap detection module, which is used to detect the box gap length of the target box under the double box lifting tool, and determine the extension length value of the lock in the lifting tool based on the detection result;

[0009] The control module is communicatively connected to the container number identification module, the layer height scanning module, and the container gap detection module. It is used to acquire the container number identification result, the height information of the low-shell container and the height information of the high-shell container, and the telescopic length value of the lock in the lifting device, and generate a single container operation instruction based on the acquired information.

[0010] The work instruction conversion module is paired with the single-box operation production system through the TOS interface. According to the pairing rules, the single-box operation instruction is converted into a double-box operation instruction. When a double-box operation abnormality occurs, the double-box operation instruction is converted into a work instruction or directly transferred to manual control.

[0011] A visualization display module, which is connected to the work instruction conversion module, is used to display the work instructions and the work exceptions to the workers in a visual manner.

[0012] Furthermore, the container number recognition module includes two sets of container number recognition vision cameras installed at both ends of the spreader. To ensure the best recognition effect, the container number recognition vision cameras are respectively installed at the ends of the spreader at diagonal positions, which can simultaneously recognize the container numbers of the front and rear containers and match the recognized container numbers with the operation instruction information.

[0013] Furthermore, the floor height scanning module includes floor height scanners symmetrically distributed on both sides of the spreader. The floor height scanners scan and detect the floor height of the two boxes and store the height information of the lower and higher boxes. Before operation, if the floor height information is found to be inconsistent with the system site map information, the operation is stopped to avoid safety risks and prevent damage to the bowling rack. When the floor heights of the higher and lower boxes are the same, if the height difference between the higher and lower boxes is detected to be greater than the set safety height difference, the operation is stopped to avoid safety risks.

[0014] Furthermore, the floor height scanner is a 2D scanner or a multi-line scanner. To ensure the best scanning results, when the floor height scanner is a 2D scanner, there are four of them; when the floor height scanner is a multi-line scanner, there are two of them.

[0015] Furthermore, the box gap detection module includes a gap scanner and four gap vision cameras. By arranging two redundant schemes, the gap scanner and the vision detection, the detection results are cross-checked, thereby achieving basic alignment of the center lock in the X direction.

[0016] Furthermore, to ensure optimal scanning results, the gap scanner is installed at the center of the bottom of the lifting device, and the gap vision cameras are symmetrically distributed at the lower end of the lifting device.

[0017] Furthermore, the box gap detection module also includes a data analysis module and a data storage module. The data storage module is used to store the detection data of the gap scanner and the four gap vision cameras. The data analysis module is used to analyze the data collected by the gap scanner and the gap vision cameras to obtain the accurate value of the box gap.

[0018] Furthermore, the specific implementation steps of the box gap detection module are as follows:

[0019] Step 1: Scan the gap length between adjacent target boxes below the lifting device at time t using the gap scanner, and record the scanning result value as xn;

[0020] Step 2: Scan the length of the gap between adjacent target boxes below the lifting device at time t using the vision camera, and record the scanning result value as yn;

[0021] Step 3: Obtain the values ​​xn and yn through the data analysis module, calculate the difference between xn and yn at time t, and denot it as Δn;

[0022] Step 4: Compare the value of △n with the preset safety threshold σ through the data analysis module. If △n≤σ, take the average of xn and yn as the detection output result. The output result is (xn+yn) / 2.

[0023] Step 5: If Δn > σ in step 4, the data analysis module performs a comprehensive judgment on the values ​​of xn and yn respectively, and the judgment result is used as the detection output result.

[0024] Furthermore, the comprehensive determination method includes the following steps:

[0025] Step 1) Read all the scan result values ​​xn and yn stored in the data storage module before time t, arrange the result values ​​in the order of acquisition time, and obtain the result sets of the gap scanner and the vision camera, respectively denoted as (x1, x2, x3, ... xn) and (y1, y2, y3, ... yn);

[0026] Step 2): Based on the result set, calculate the failure rate λ of the gap scanner and the visual camera. x and λ y The calculation formulas are as follows: Where n is the total number of data samples in the set, and λ1 and λ2 are the number of detected failure samples in the sets (x1, x2, x3, ... xn) and (y1, y2, y3, ... yn), respectively;

[0027] Step 3), based on the failure rate λ in step 2). x and λ y The failure rate calculation results are then substituted into the formula to calculate the reliability R of the gap scanner and the vision camera at time t. x and R y The reliability calculation formula is: R(t) = e -λt ;

[0028] Step 4), compare the reliability R of the gap scanner and the vision camera. x and R y The value of the reliability value is taken as the detection output result of the system with the larger reliability value.

[0029] Furthermore, in step 2), the detected failure sample is a scan result value whose deviation from the actual value exceeds a set value, which is manually set by the operator based on actual operating experience.

[0030] The present invention has the following beneficial effects:

[0031] 1) The technical solution proposed in this invention, by setting up a container number recognition module, uses AI artificial intelligence recognition algorithm to recognize the container number on the top of the container and the container number on the door, and generates the container number recognition result. It can simultaneously recognize the container numbers of the front and rear containers, and match the recognized container numbers with the instruction information. This solves the problem of relying on manual observation and confirmation of container numbers in the original production system, improves the accuracy of recognition, and realizes automated production operations.

[0032] 2) The technical solution proposed in this invention, by setting up a layer height scanning module, scans and detects the layer height of the double-shell container and stores the height information of the lower shell container and the upper shell container. Before operation, if the layer height information of the operation is found to be inconsistent with the system field map information, the operation is stopped to avoid safety risks and prevent damage to the container. When the layer heights of the upper and lower shell containers are the same, if the height difference between the upper and lower shell containers is detected to be greater than the set safety height difference, the operation is stopped to avoid safety risks. This can effectively solve the layer height problem of double-shell operation, improve operation safety, reduce operation risks, and the scanning accuracy can reach between 3cm and 5cm. It can effectively solve the problem of container height difference and avoid problems such as damage to lifting equipment and container smashing caused by container height difference.

[0033] 3) The technical solution proposed in this invention sets up a box gap detection module and arranges two redundant schemes, a gap scanner and a visual inspection, to cross-verify the detection results, thereby achieving the basic alignment of the center lock in the X direction. This solves the problem that in the traditional operation mode, when the double box spreader is used to grab and place boxes in the lane and box area, the operator needs to manually adjust the size of the center lock gap of the double boxes to complete the operation. In the process of automated double box operation, it is necessary to automatically detect the gap between the boxes at the target position of the double boxes, so as to realize the automatic matching of the center lock gap of the double box spreader with the gap at the target position to complete the box placement operation process.

[0034] 4) The technical solution proposed in this invention sets up a work instruction conversion module, which pairs with the single-box operation production system through the TOS interface. According to the pairing rules, the 20-foot single box that meets the requirements for double-box operation is converted into a double-box instruction, and the data interaction mode is consistent with that of the single-box operation instruction. At the same time, it meets the functions of instruction acquisition, release, confirmation, and abnormal message acquisition. When a double-box operation abnormality occurs, the double-box operation instruction is converted into a work instruction or directly transferred to manual control, thus solving the problem of low support for double-box operation tasks in the original production system. Attached Figure Description

[0035] Figure 1This is an overall block diagram of the port RMG or RTG dual-container automated operation system based on visual situational awareness of the present invention.

[0036] Figure 2 This is a schematic diagram of one installation structure of the box number recognition visual camera of the present invention;

[0037] Figure 3 This is a schematic diagram of another installation structure for the box number recognition visual camera of the present invention;

[0038] Figure 4 This is a schematic diagram of the installation structure of the layer height scanner of the present invention;

[0039] Figure 5 This is a schematic diagram of the installation structure of the gap scanner of the present invention;

[0040] Figure 6 This is a schematic diagram of the installation structure of the gap vision camera of the present invention;

[0041] Figure 7 This is a flowchart of the operation process control logic of the operation instruction conversion module of the present invention;

[0042] Figure 8 This is a block diagram of the operation process control logic of the operation instruction conversion module of the present invention under abnormal conditions. Detailed Implementation

[0043] The present invention will be further described below with reference to specific embodiments. The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the present invention. In order to better illustrate the specific embodiments of the present invention, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the actual product size.

[0044] Example 1

[0045] like Figure 1 As shown, a port RMG or RTG dual-container automated operation system based on visual situational awareness includes:

[0046] ① Container number recognition module: The container number recognition module uses AI artificial intelligence recognition algorithm to identify the container number on the top of the container and the container number on the door, and generate the container number recognition result;

[0047] Taking the CRNN neural network algorithm as an example, its principle is as follows:

[0048] First, a CNN network extracts convolutional feature maps from the input image, transforming the (32, 100, 3) image into a (1, 25, 512) convolutional feature matrix. Then, an LSTM network further extracts text sequence features based on the convolutional features, using a stacked deep bidirectional structure. Since the CNN output feature map is (1, 25, 512), the maximum time length for the RNN is T = 25 (i.e., there are 25 time inputs, each with a column vector D = 512). Finally, Transcription Layers apply softmax to the RNN output to produce the character output.

[0049] The container number recognition module includes two sets of container number recognition vision cameras installed at both ends of the spreader, such as... Figure 2 and Figure 3 As shown, the box number recognition vision cameras are installed at the ends of the diagonal positions of the lifting device;

[0050] During the operation of the yard crane, the visual camera acquires images of the container numbers on the side and top of the container based on the real-time status of the operation, performs real-time identification, and outputs the identified information.

[0051] By setting up a container number recognition module, an AI artificial intelligence recognition algorithm is used to identify the container number on the top of the container and the container number on the door, and generate the container number recognition result. It can simultaneously identify the container numbers of the front and rear containers, and match the identified container numbers with the instruction information. This solves the problem of relying on manual observation and confirmation of container numbers in the original production system, improves the accuracy of recognition, and realizes automated production operations.

[0052] ② Layer height scanning module: The layer height scanning module is used to scan the adjacent shell boxes of the trolley, obtain and store the height information of the low shell box and the height information of the high shell box;

[0053] The floor height scanning module includes floor height scanners symmetrically distributed on both sides of the hanger; the floor height scanners are 2D scanners or multi-line scanners, such as... Figure 4 As shown, when the floor height scanner is a 2D scanner, there are four of them; when the floor height scanner is a multi-line scanner, there are two of them.

[0054] By setting up a layer height scanning module, the layer height of the double-shell container is scanned, detected, and stored. The height information of the lower and higher shell containers is stored. Before operation, if the layer height information is found to be inconsistent with the system site map information, the operation is stopped to avoid safety risks and prevent damage to the container. When the layer heights of the higher and lower shell containers are the same, if the height difference between the higher and lower shell containers is detected to be greater than the set safety height difference, the operation is stopped to avoid safety risks. This can effectively solve the layer height problem in double-shell container operations, improve operational safety, and reduce operational risks. Moreover, the scanning accuracy can reach between 3cm and 5cm, which can effectively solve the problem of container height difference and avoid problems such as damage to lifting equipment and container smashing caused by container height difference.

[0055] ③ Box gap detection module: The box gap detection module is used to detect the box gap length of the target box under the double box lifting device, and determine the extension length value of the lock in the lifting device based on the detection results;

[0056] The box gap detection module includes a gap scanner and four gap vision cameras; such as Figure 5 and Figure 6 As shown, the gap scanner is installed at the center of the bottom of the lifting device, and the gap vision cameras are symmetrically distributed at the lower end of the lifting device.

[0057] The box gap detection module also includes a data analysis module and a data storage module. The data storage module is used to store the detection data from the gap scanner and four gap vision cameras, and the data analysis module is used to analyze the data collected by the gap scanner and gap vision cameras to obtain the accurate value of the box gap.

[0058] The specific implementation steps of the box gap detection module are as follows:

[0059] Step 1: Scan the gap length between adjacent target boxes below the spreader at time t using a gap scanner, and record the scanning result value as xn;

[0060] Step 2: Scan the length of the gap between adjacent target boxes below the lifting device at time t using a vision camera, and record the scan result value as yn;

[0061] Step 3: Obtain the values ​​xn and yn through the data analysis module, calculate the difference between xn and yn at time t, and denot it as Δn;

[0062] Step 4: Compare the value of △n with the preset safety threshold σ through the data analysis module. If △n≤σ, take the average of xn and yn as the detection output result. The output result is (xn+yn) / 2.

[0063] Step 5: If Δn > σ in step 4, the data analysis module will comprehensively determine the values ​​of xn and yn respectively, and the determination result will be used as the detection output result.

[0064] The comprehensive judgment method includes the following steps:

[0065] Step 1) Read all the scan result values ​​xn and yn stored in the data storage module before time t, arrange the result values ​​in the order of acquisition time, and obtain the result sets of the gap scanner and vision camera, which are denoted as (x1, x2, x3, ... xn) and (y1, y2, y3, ... yn) respectively;

[0066] Step 2), based on the result set, calculate the failure rate λ of the gap scanner and the visual camera. x and λ y The calculation formulas are as follows: Where n is the total number of data samples in the set, and λ1 and λ2 are the number of detected failure samples in the sets (x1, x2, x3, ... xn) and (y1, y2, y3, ... yn), respectively;

[0067] Step 3), based on the failure rate λ in step 2). x and λ y The calculation results, substituted into the failure rate calculation results, are used to calculate the reliability R of the time-slot scanner and vision camera at time t. x and R y The formula for calculating reliability is: R(t) = e -λt ;

[0068] Step 4), compare the reliability of the gap scanner and the vision camera. x and R y The value of the reliability value is taken as the detection output result of the system with the larger reliability value.

[0069] By introducing the concept of reliability, the system's operational reliability is used to determine when two sets of systems produce inconsistent test outputs. By selecting data collected by a more reliable device as the test output, the problem of data deviation caused by equipment failure can be avoided to some extent.

[0070] The failed sample detected in step 2) is the scan result value whose deviation from the actual value exceeds the set value. The set value is manually set by the operator based on actual operating experience.

[0071] By setting up a box gap detection module and deploying two redundant schemes of gap scanner and visual inspection, the detection results are cross-verified, thereby achieving basic alignment of the center lock in the X direction. This solves the problem that in the traditional operation mode, when the double box spreader is used to grab and place boxes in the lane and box area, the operator needs to manually adjust the size of the center lock gap of the double boxes to complete the operation. In the process of automated double box operation, it is necessary to automatically detect the gap between the boxes at the target position of the double boxes, so as to achieve automatic matching between the center lock gap of the double box spreader and the target position gap to complete the box placement operation process.

[0072] ④ Job instruction conversion module, such as Figure 7 and Figure 8 As shown, the operation instruction conversion module pairs with the single-container operation production system through the TOS interface. According to the pairing rules, it converts the single-container operation instruction into a double-container operation instruction. The pairing rules include integrating and processing key information such as the two 20-foot containers, the same site, the same column, the same layer, the same vehicle, adjacent bays, container weight, container type, container type, and cargo attributes to generate a double-container operation instruction. When a double-container operation anomaly occurs, the double-container instruction can be converted to a single-container instruction or directly converted to manual control to avoid the yard crane being unable to operate when the double-container operation instruction cannot be implemented, as shown in Figure X. When a double-container operation anomaly occurs, the double-container operation instruction is converted into an operation instruction or directly converted to manual control.

[0073] By setting up a work instruction conversion module and pairing it with the single-box operation production system via the TOS interface, the module converts 20-foot single-box operations that meet the pairing rules into double-box instructions, maintaining consistency with the data interaction mode of single-box operation instructions. It also fulfills functions such as instruction acquisition, release, confirmation, and exception message acquisition. Furthermore, when a double-box operation exception occurs, the double-box operation instruction is converted back into a work instruction or directly transferred to manual control, thus resolving the original production system's limited support for double-box operation tasks.

[0074] ⑤ Control module: The control module is connected to the container number recognition module, the layer height scanning module and the container gap detection module. It is used to obtain the container number recognition result, the height information of the low-shell container and the height information of the high-shell container, and the extension length value of the lock in the lifting device. Based on the obtained information, it generates a single container operation instruction.

[0075] ⑥ Visualization module: The visualization module is connected to the work instruction conversion module and is used to display work instructions and work exceptions to the workers in a visual manner.

[0076] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A port RMG or RTG dual-container automated operation system based on visual situational awareness, characterized in that: The operating system includes: The container number recognition module uses an AI artificial intelligence recognition algorithm to identify the container number on the top of the container and the container number on the door, and generates the container number recognition result. The layer height scanning module is used to scan the adjacent shell boxes of the trolley, acquire and store the height information of the low shell box and the height information of the high shell box; Box gap detection module, which is used to detect the box gap length of the target box under the double box lifting tool, and determine the extension length value of the lock in the lifting tool based on the detection result; The control module is communicatively connected to the container number identification module, the layer height scanning module, and the container gap detection module. It is used to acquire the container number identification result, the height information of the low-shell container and the height information of the high-shell container, and the telescopic length value of the lock in the lifting device, and generate a single container operation instruction based on the acquired information. The work instruction conversion module is paired with the single-box operation production system through the TOS interface. According to the pairing rules, the single-box operation instruction is converted into a double-box operation instruction. When a double-box operation abnormality occurs, the double-box operation instruction is converted into a work instruction or directly transferred to manual control. A visualization display module, which is connected to the work instruction conversion module, is used to display the work instructions and the work anomalies to the workers in a visual manner. The box gap detection module includes a gap scanner and four gap vision cameras; The box gap detection module also includes a data analysis module and a data storage module. The data storage module is used to store the detection data of the gap scanner and the four gap vision cameras. The data analysis module is used to analyze the data collected by the gap scanner and the gap vision cameras to obtain the accurate value of the box gap. The specific implementation steps of the box gap detection module are as follows: Step 11: Scan the gap length between adjacent target boxes below the lifting device at time t using the gap scanner, and record the scanning result value as xn; Step 12: Scan the gap length between adjacent target boxes below the lifting device at time t using the vision camera, and record the scan result value as yn; Step 13: Obtain the values ​​xn and yn through the data analysis module, calculate the difference between xn and yn at time t, and denot it as Δn; Step 14: Compare the value of △n with the preset safety threshold σ through the data analysis module. If △n≤σ, take the average of xn and yn as the detection output result. The output result is (xn+yn) / 2. Step 15: If Δn > σ in step 14, the data analysis module performs a comprehensive judgment on the values ​​of xn and yn respectively, and the judgment result is used as the detection output result. The comprehensive determination method includes the following steps: Step 21: Read all the scan result values ​​xn and yn stored in the data storage module before time t, arrange the result values ​​in the order of acquisition time, and obtain the result sets of the gap scanner and the vision camera, respectively denoted as (x1, x2, x3, ... xn) and (y1, y2, y3, ... yn); Step 22: Calculate the failure rate of the gap scanner and the visual camera based on the result set. and The calculation formulas are as follows: Where n is the total number of data samples in the set. and These represent the number of failed detection samples in the sets (x1, x2, x3, ..., xn) and (y1, y2, y3, ..., yn), respectively. Step 23, based on the failure rate in step 22 and The failure rate calculation results are then substituted into the formula to calculate the reliability R of the gap scanner and the vision camera at time t. x and R y The formula for calculating the reliability is as follows: ; Step 24: Compare the reliability R of the gap scanner and the vision camera. x and R y The value of the reliability value is taken as the detection output result of the system with the larger reliability value.

2. The port RMG or RTG dual-container automated operation system based on visual situational awareness according to claim 1, characterized in that: The container number recognition module includes two sets of container number recognition vision cameras installed at both ends of the spreader, with the container number recognition vision cameras respectively installed at the ends of the spreader at diagonal positions.

3. The port RMG or RTG dual-container automated operation system based on visual situational awareness according to claim 1, characterized in that: The floor height scanning module includes floor height scanners symmetrically distributed on both sides of the lifting device.

4. The port RMG or RTG dual-container automated operation system based on visual situational awareness according to claim 3, characterized in that: The floor height scanner is a 2D scanner or a multi-line scanner. When the floor height scanner is a 2D scanner, there are four of them; when the floor height scanner is a multi-line scanner, there are two of them.

5. The port RMG or RTG dual-container automated operation system based on visual situational awareness according to claim 1, characterized in that: The gap scanner is installed at the center of the bottom of the lifting device, and the gap vision cameras are symmetrically distributed at the lower end of the lifting device.

6. The port RMG or RTG dual-container automated operation system based on visual situational awareness according to claim 1, characterized in that: The failed sample detected in step 22 is a scan result value whose deviation from the actual value exceeds a set value, which is manually set by the operator based on actual operating experience.

Images