A single- and double-tank detection method and device for quay crane based on machine vision

Through the single and double box detection method of shore bridge based on machine vision, the container image is detected in real time using neural network, and combined with the number of on-board containers and text information, the misjudgment problem caused by inaccurate image acquisition in the existing technology is solved, and the single The accuracy of double-box judgment and the reliability of box hole detection.

CN114119496BActive Publication Date: 2024-11-01BEIJING AEROSPACE AUTOMATIC CONTROL RES INST
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202111291272.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-29
Publication Date
2024-11-01
Estimated Expiration
2041-10-29

AI Technical Summary

Technical Problem

In the prior art, there are problems of misjudgment and inaccurate data acquisition through box hole identification through container intermediate images, resulting in low accuracy of double-box misjudgment and judgment.

Method used

Using the single and double box detection method of shore bridge based on machine vision, a neural network Yolov5 is established by obtaining multiple container images, and the container image is detected in real time using the object detection model. Combining the number of vehicles on board, the box hole and text information, the total score is calculated. Judge odd and even boxes to ensure the accuracy of image acquisition.

Benefits of technology

The accuracy of single and double box judgment is improved, and misjudgment due to single information judgment is avoided, which enhances the accuracy of box hole detection and data acquisition reliability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114119496B_ABST
    Figure CN114119496B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for detecting single and double containers of a quay crane based on machine vision, which belongs to the technical field of port safety operation assistance, and solves the problem that the intermediate image obtained by the existing method is not necessarily the middle area of ​​the container and the presence or absence of the container hole in the intermediate image leads to misjudgment of the container hole. The method includes obtaining multiple container images from a database and annotating each container image to generate a container image set; establishing a neural network Yolov5 and training the neural network Yolov5 using the container image set to obtain a target detection model; using the target detection model to perform real-time detection on the container image to be tested to obtain a detection result, and the detection result includes: the number of containers on the vehicle, the box holes and the text of the container on the vehicle; and judging whether the container on the vehicle is a single box or a double box based on the detection result. The single and double boxes are judged jointly by the number of containers, the box holes and the text of the container to effectively improve the accuracy of the judgment of single and double boxes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of port safety operation assistance, and in particular to a machine vision-based quay crane single and double container detection method and device. Background Art

[0002] Containers are important carriers in modern logistics and transportation. The efficiency of container loading and unloading operations at ports directly affects the efficiency of the entire logistics and transportation. When grabbing containers on container trucks, the quay crane will adjust the spreader shape according to the single or double container situation on the current operating vehicle to adapt to different operating conditions. Traditional manual visual inspection methods are not only inefficient, but also inevitably lead to misjudgments due to fatigue and negligence of personnel, which in turn cause accidents.

[0003] In the prior art, the gap between boxes is detected by using photoelectric switches. However, since this method requires a large number of photoelectric sensors, the internal circuit is complicated and is easily disturbed by the external environment, resulting in erroneous judgment.

[0004] The machine vision part of the existing double-box detection method uses a camera to shoot the middle of the container, thereby obtaining an image of the middle of the container, and uses a box hole recognition model to identify the box holes, and then determine whether it is a single or double box.

[0005] The existing methods have the following problems:

[0006] 1. This method only relies on the hole recognition of the middle image of the container to determine whether there are double containers. It does not deal with missed or false detection of the hole, so there is a risk of false judgment of double containers, which reduces the accuracy of judgment.

[0007] 2. This method cannot guarantee the accuracy of data collection, that is, it cannot guarantee that the collected image is exactly the middle area of ​​the container, which may lead to misjudgment. Summary of the invention

[0008] In view of the above analysis, an embodiment of the present invention aims to provide a method and device for detecting single and double containers on a quay crane based on machine vision, so as to solve the problem that the intermediate image obtained by the existing method cannot be guaranteed to be an image of the middle area of ​​the container and the presence or absence of container holes in the intermediate image leads to misjudgment of container holes.

[0009] On the one hand, an embodiment of the present invention provides a method for detecting single and double containers on a quay crane based on machine vision, comprising: acquiring multiple container images from a database and annotating each container image to generate a container image set; establishing a neural network Yolov5 and training the neural network Yolov5 using the container image set to obtain a target detection model; using the target detection model to perform real-time detection on the container image to be tested to obtain a detection result, wherein the detection result includes; the number of on-board containers, box holes and on-board container text; and judging whether the on-board container is a single box or a double box based on the detection result.

[0010] The beneficial effects of the above technical solution are as follows: the detection results include the number of containers on the vehicle, the box holes and the text on the containers on the vehicle. The above detection results are used to determine whether there are single or double containers, thereby avoiding misjudgment caused by relying on single information to determine whether there are single or double containers.

[0011] Based on a further improvement of the above method, judging whether the vehicle-mounted container is a single box or a double box based on the detection result further includes: calculating a total score according to whether the container area in the container image is two 20 feet, combined with the box hole and text detection information of the middle sub-area, wherein when the calculated total score is greater than a threshold, judging that the vehicle-mounted container is a double box; and when the calculated total score is less than or equal to a threshold, judging that the vehicle-mounted container is a single box.

[0012] Based on the further improvement of the above method, the total score is calculated by the following formula:

[0013]

[0014] Among them, weight0 represents the weight of the number of containers, whether weight1 has a text weight, and whether weight2 has a box hole weight. R0 represents whether there are two containers, whether R1 has text, and whether R2 has a box hole. If yes, it is 1, and if no, it is 0.

[0015] The beneficial effects of the above technical solution are as follows: by fusing the detected number of on-board containers, box holes and on-board container text, the accuracy of single and double box judgment is improved through the fused detection results, and there is no need to deal with missed or false detection of box holes.

[0016] Based on further improvement of the above method, the machine vision-based quay crane single and double box detection method further includes: transmitting the judgment result of single box or double box to the quay crane control system; when grabbing the vehicle container on the container truck, according to whether the vehicle container on the container truck is a single box or a double box, the quay crane control system adjusts the sling shape of the quay crane to grab a single vehicle container or two vehicle containers.

[0017] Based on the further improvement of the above method, obtaining multiple container images from a database and annotating each container image to generate a container image set further includes: obtaining multiple historical container images from a database or obtaining container images to be tested in real time; using the Labelimg tool to annotate the containers in each historical container image to establish a first data set; after container detection, randomly cropping highly determined containers in the historical container images or the container images to be tested to generate historical container sub-images or container sub-images to be tested; and using the Labelimg tool to annotate the container holes and text in the historical container sub-images to establish a second data set.

[0018] Based on further improvements of the above method, establishing a neural network Yolov5 and training the neural network Yolov5 using the container image set to obtain a target detection model further includes: establishing a first neural network Yolov5 and training the first neural network Yolov5 using a part of the first data set to obtain a first target detection model; using the first target detection model to perform container detection on another part of the first data set to distinguish the height of the container as 20 feet or non-20 feet; and after establishing the second data set, establishing a second neural network Yolov5 and training the second neural network Yolov5 using the second data set to obtain a second target detection model.

[0019] Based on a further improvement of the above method, randomly cropping highly determined containers in the historical container image or the container image to be tested to generate a historical container sub-image or a container sub-image to be tested further includes: merging two minimum circumscribed rectangles of containers on the same container truck into a single minimum circumscribed rectangle according to the height and width of the minimum circumscribed rectangles of any two 20-foot containers in the historical container image or the container image to be tested; determining the cropping ratio of the container according to the position of the center point of the fused single minimum circumscribed rectangle in the image, and randomly cropping the historical container image or the container image to be tested into an upper sub-region image, a middle sub-region image and a lower sub-region image according to the cropping ratio, wherein the middle sub-region image is subjected to box hole and text detection.

[0020] Based on a further improvement of the above method, using the target detection model to perform real-time detection on the container image to be tested to obtain the detection result further includes: using the first target detection model to perform container detection on the container image to be tested to distinguish the height of the container in the container image to be tested as 20 feet or non-20 feet; and using the second target detection model to detect the container sub-image to be tested to identify the box holes and the on-board container text in the container sub-image to be tested.

[0021] Based on a further improvement of the above method, obtaining multiple container image sets further includes: installing a first camera and a second camera at opposite sides of a lane on a quay crane beam; using the first camera and the second camera to capture images of containers on a container truck; and storing the captured container images in a database as historical container images and using the container images collected in real time as the container images to be tested.

[0022] On the one hand, an embodiment of the present invention provides a single- and double-box detection device for a quay crane based on machine vision, comprising: a container image set generation module, used to obtain multiple container images from a database and annotate each container image to generate a container image set; a model generation module, used to establish a neural network Yolov5 and train the neural network Yolov5 using the container image set to obtain a target detection model; a detection module, used to use the target detection model to perform real-time detection on the container image to be tested to obtain a detection result, wherein the detection result includes; the number of on-board containers, box holes and on-board container text; and a judgment module, used to judge whether the on-board container is a single box or a double box based on the detection result.

[0023] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0024] 1. By fusing the detected number of containers, container holes and container text on the vehicle, and using the fused detection results to judge whether a container is single or double, the accuracy of judgment on single or double containers is improved. Therefore, there is no need to deal with missed or false detection of container holes, thus avoiding misjudgment caused by relying on single information to judge whether a container is single or double.

[0025] 2. By first determining the location of the container area, the image of the box seam area can be more effectively obtained, thereby effectively improving the accuracy of single and double box discrimination.

[0026] 3. According to the position of the center point of the detected rectangular area in the image, the height ratio of the container image cropping is adjusted to ensure that the middle sub-area image is exactly the middle area of ​​the container.

[0027] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0029] Figure 1 The figure is a flow chart of a method for detecting single and double boxes of a quay crane based on machine vision according to an embodiment of the present invention.

[0030] Figure 2 is a schematic diagram of an installation arrangement of a camera device according to an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of container detection and fusion according to an embodiment of the present invention;

[0032] Figure 4 is a specific flow chart of a method for detecting single and double containers of a quay crane based on machine vision according to an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of container region cropping and fusion in a container image according to an embodiment of the present invention; and

[0034] Figure 6 It is a schematic diagram of a single and double-tank detection device for a quay crane based on machine vision according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0036] A specific embodiment of the present invention discloses a method for detecting single and double containers on a quay crane based on machine vision. Figure 1The machine vision-based single and double container detection method for a quay crane includes: in step S102, a plurality of container images are obtained from a database and each container image is annotated to generate a container image set; in step S104, a neural network Yolov5 is established and trained using the container image set to obtain a target detection model; in step S106, the target detection model is used to perform real-time detection on the container image to be tested to obtain a detection result, wherein the detection result includes: the number of containers on the vehicle, the box holes and the text of the containers on the vehicle; and in step S108, based on the detection result, it is determined whether the container on the vehicle is a single box or a double box.

[0037] Compared with the prior art, the machine vision-based quay crane single and double container detection method provided in this embodiment has detection results including the number of containers on board, container holes and text on the containers on board. Single and double container judgment is performed based on the above detection results, thereby avoiding misjudgment caused by relying on single information to determine single and double containers.

[0038] In the following, reference will be made to Figures 1 to 5 , steps S102 to S108 in the machine vision-based quay crane single and double box detection method are described in detail.

[0039] First, a plurality of container image sets are obtained. The plurality of container image sets further include: installing a first camera and a second camera at opposite sides of a lane on a quay crane beam; using the first camera and the second camera to capture images of containers on a container truck; and storing the captured container images in a database as historical container images and using the real-time captured container images as container images to be tested.

[0040] In step S102, a plurality of container images are obtained from a database and each container image is annotated to generate a container image set. Specifically, obtaining a plurality of container images from a database and annotating each container image to generate a container image set further includes: obtaining a plurality of historical container images from a database or obtaining a container image to be tested in real time; annotating the containers in each historical container image using a Labelimg tool to establish a first data set; after container detection, randomly cropping highly determined containers in the historical container image or the container image to be tested to generate a historical container sub-image or a container sub-image to be tested; and annotating the container holes and text in the historical container sub-image using a Labelimg tool to establish a second data set.

[0041] Specifically, randomly cropping the highly determined containers in the historical container image or the container image to be tested to generate the historical container sub-image or the container sub-image to be tested further includes: according to the minimum circumscribed rectangle of any two 20-foot containers in the historical container image or the container image to be tested (reference Figure 3 The height and width of the two thin white solid rectangles of each container truck are combined into a single minimum enclosing rectangle (reference Figure 3 The black thick solid rectangle of each container truck in the image); according to the position of the center point of the fused single minimum circumscribed rectangle in the image, the cropping ratio of the container is determined, and the historical container image or the container image to be tested is randomly cropped into the upper sub-region image and the middle sub-region image according to the cropping ratio (reference Figure 3 The thin black solid rectangle on each set card in the Figure 5 ), wherein the upper sub-region image and the lower sub-region image are subjected to box hole detection, and the middle sub-region image is subjected to box hole and text detection to determine the box hole coordinates, wherein the box hole analysis is performed according to the box hole coordinates to determine whether the box hole is detected. Specifically, when the Y coordinate of the center point of the detection rectangular area is less than 1 / 3 of the image height, the cropping ratio is 0.15, 0.4, 0.45, when the Y coordinate is greater than 1 / 3 and less than 2 / 3, the ratio is 0.25, 0.4, 0.35, and when the Y coordinate is greater than 2 / 3, the ratio is 0.35, 0.4, 0.25. Specifically, the middle sub-region image has a fixed image ratio, and the center of the entire image is shifted up and down as needed to ensure that the box hole and text can be detected.

[0042] In step S104, a neural network Yolov5 is established and trained using the container image set to obtain a target detection model. The target detection model includes a first target detection module and a second target detection module. Specifically, establishing a neural network Yolov5 and training the neural network Yolov5 using the container image set to obtain a target detection model further includes: establishing a first neural network Yolov5 and training the first neural network Yolov5 using a part of the first data set to obtain a first target detection model; using the first target detection model to perform container detection on another part of the first data set to distinguish the height of the container as 20 feet or 40 feet. Establishing a second neural network Yolov5 and training the second neural network Yolov5 using the second data set established in step S102 to obtain a second target detection model.

[0043] In step S106, the target detection model is used to perform real-time detection on the container image to be tested to obtain a detection result, wherein the detection result includes: the number of containers on the vehicle, the container holes and the container text on the vehicle. The real-time detection of the container image to be tested using the target detection model to obtain the detection result further includes: using the first target detection model to perform container detection on the container image to be tested to distinguish the height of the container as 20 feet or non-20 feet, that is, 40 feet, that is, to detect whether the number of containers is two; and using the second target detection model to detect the container sub-image to be tested to identify the container holes and the container text on the vehicle in the container sub-image to be tested.

[0044] In step S108, it is determined whether the vehicle-mounted container is a single container or a double container based on the detection result. Specifically, the determination of whether the vehicle-mounted container is a single container or a double container based on the detection result further includes: calculating the total score based on whether the container area in the container image is two 20-foot containers, combined with the container hole and text detection information of the middle sub-area. The total score is calculated by the following formula:

[0045]

[0046] Among them, weight0 represents the weight of the number of containers, whether weight1 has a text weight, and whether weight2 has a box hole weight. R0 represents whether there are two containers, whether R1 has text, and whether R2 has a box hole. If yes, it is 1, and if no, it is 0.

[0047] When the total score calculated by the above formula is greater than the threshold, the vehicle-mounted container is judged to be a double container; and when the total score calculated by the above formula is less than or equal to the threshold, the vehicle-mounted container is judged to be a single container.

[0048] After determining whether the vehicle-mounted container is a single container or a double container based on the detection result, the method further includes: transmitting the determination result of the single container or the double container to the quay crane control system; when grabbing the vehicle-mounted container on the container truck, according to whether the vehicle-mounted container on the container truck is a single container or a double container, the quay crane control system adjusts the sling shape of the quay crane to grab a single vehicle-mounted container or two vehicle-mounted containers.

[0049] Another specific embodiment of the present invention discloses a single- and double-tank detection device for a quay crane based on machine vision. Figure 6The machine vision-based quay crane single- and double-box detection device includes: a container image set generation module 602, which is used to obtain multiple container images from a database and annotate each container image to generate a container image set; a model generation module 604, which is used to establish a neural network Yolov5 and train the neural network Yolov5 using the container image set to obtain a target detection model; a detection module 606, which is used to use the target detection model to perform real-time detection on the container image to be tested to obtain a detection result, wherein the detection result includes; the number of on-board containers, box holes and on-board container text; and a judgment module 608, which is used to judge whether the on-board container is a single box or a double box based on the detection result.

[0050] In the following, reference Figures 2 to 5 , by means of specific examples, the single and double box detection method of the quay crane based on machine vision is described in detail.

[0051] The technical problem to be solved by the present invention is to first detect and identify the rectangular area of ​​the container on the container truck, obtain the effective container intermediate image area based on the analysis and judgment of the position and size of the rectangular area, and comprehensively judge whether it is an odd or even container by combining the container detection result, the text detection result on the container, and the number and distance information of the box holes, thereby effectively improving the accuracy of odd and even container discrimination.

[0052] According to the machine vision-based single and double container detection method of the quay crane in the embodiment of the present invention, intelligent video analysis technology is used to effectively identify the container area, container holes, text and other information on the container truck, and analyze and distinguish them to achieve the determination of single and double containers of the quay crane container truck. Figure 4 , the method adopts the following technical solution:

[0053] 1. Reference Figure 2 Two cameras are installed on the crossbeam of the quay crane to collect container data on the operating container trucks.

[0054] 2. Collect container data on the container truck, use the labelimg tool to mark the containers, container holes and container text, and then train the corresponding target detection algorithm to realize the detection and recognition of containers, container holes and text.

[0055] 3. Reference Figure 5 When a vehicle loaded with containers enters the camera field of view, the container on the truck is first identified through the target recognition algorithm, and the size is determined by identifying the container area to distinguish between 20-foot and non-20-foot containers. Then, according to certain conditions, the 20-foot boxes on the same truck are fused to obtain the minimum circumscribed rectangle of the area of ​​the two 20-foot containers.

[0056] 4. Obtain the cropping ratio of the area according to the position of the center coordinates of the fused bounding rectangle or the bounding rectangle of the single box in the image.

[0057] 5. According to the above ratio, the detected inner container area of ​​the container truck is cropped. For the container truck loaded with containers, the cropped area is divided into three sub-area images, and the box holes and text are detected in the middle cropped area.

[0058] 6. According to whether the container truck area currently loaded with containers is a fusion area, that is, whether it is two 20-foot containers, combined with the box hole and text detection information of the middle sub-area, determine whether it is a single or double container, as follows:

[0059]

[0060] weight0 indicates the weight of the number of containers, weight1 indicates whether there is a text weight, and weight2 indicates whether there is a box hole weight. R0 indicates whether there are two containers, whether R1 has text, and whether R2 has a box hole. If yes, it is 1, and if no, it is 0. When the total score is greater than a certain threshold, it is determined to be a double container, otherwise it is a single container.

[0061] The present invention provides a single-box and double-box discrimination method based on machine vision, which can be divided into two parts:

[0062] The first part is to collect on-site image data for target detection algorithm training and verification. The specific steps are as follows:

[0063] 1. Collect the vehicle-mounted container data taken by the camera, and use labelimg to mark the vehicle-mounted containers (Note: the main marking category is container, and this process does not distinguish between container types, i.e. 20 feet or 40 feet).

[0064] 2. Use the single-stage target detection algorithm Yolov5 for model training, set the target detection algorithm input size to 640*640, and generate the target detection model Yolov5-0.

[0065] 3. Reference Figure 2 , use the target detection model generated in step 2 to detect targets in the same data set, and distinguish 20 feet or 40 feet according to the height of the detected container rectangular area, and fuse the two containers on the same container truck according to the height and width of the minimum circumscribed rectangle of any two 20-foot containers. And randomly crop the center point of the fused minimum circumscribed rectangular area within a certain range above and below, and then obtain the image of the middle area of ​​the container. At the same time, randomly crop the 40-foot container area according to a certain area height as a negative sample for training.

[0066] 4. Use labelimg to label the cropped image data. The main marking categories are the box holes and the text on the upper surface of the container.

[0067] 6. Also use the target detection algorithm Yolov5 for model training, with an input size of 512*512 to generate the target detection algorithm Yolov5-1.

[0068] Part 2: Using the recognition results of containers, container holes and text, determine whether a container is single or double, and send the results to the quay crane control system. The specific steps are as follows:

[0069] 1. First, the pre-trained target detection algorithm Yolov5-0 is used for target recognition, and then the height of the recognized rectangular area is used to determine whether it is 20 feet or not.

[0070] 2. Two 20-foot containers combined on one truck, please refer to Figure 3 That is, it is determined whether the width and height of the smallest circumscribed rectangular area of ​​any two rectangular areas determined to be 20 feet in the image are within a certain range. If the conditions are met, the two rectangular areas are merged into one area.

[0071] 3. Reference Figure 5 According to the position of the center point of the rectangular area detected in the image, the height ratio of the container image cropping is determined (the width is consistent with the width of the rectangular area). Considering that when the container area is at different positions in the image, the position of the box seam and the Y coordinate of the center point of the rectangular area will have corresponding position changes, therefore, in the present invention, when the Y coordinate of the center point of the detected rectangular area is less than 1 / 3 of the image height, the cropping ratio is 0.15, 0.4, 0.45; when the Y coordinate is greater than 1 / 3 and less than 2 / 3, the ratio is 0.25, 0.4, 0.35; when the Y coordinate is greater than 2 / 3, the ratio is 0.35, 0.4, 0.25.

[0072] 4. According to the above cropping ratio, the rectangular area of ​​the container is cropped and the image of the middle seam area is obtained.

[0073] 5. Single and double box determination, the steps are as follows:

[0074] First, the object detection model Yolov5-1 is used to perform box hole and text recognition in the middle area of ​​the image.

[0075] Secondly, the detection and recognition results are analyzed and judged. Box hole analysis: If the number of detected box holes is less than 2, it is considered that no box holes are detected. Otherwise, the detected box holes are sorted according to the X coordinate. If the minimum value after sorting and the minimum value in the X direction are within a certain threshold range, it is considered that the box hole is detected. Text analysis: If text is detected in the middle sub-area, and its rectangular height and width meet a certain range, it is considered that the text on the upper surface of the container is detected.

[0076] Finally, the single and double boxes are determined by combining the fusion information, box hole information and text information as follows:

[0077]

[0078] Among them, weight0 indicates the weight of the number of containers, for example, 0.4, weight1 indicates whether there is a text weight, for example, 0.2, and weight2 indicates whether there is a box hole weight, for example, 0.4, and R0 indicates whether there are two containers, whether R1 has text, and whether R2 has a box hole. If the total score is greater than 0.6, it is considered a double container, otherwise it is considered a single container.

[0079] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0080] 1. By fusing the number of containers, container holes and container text detected on the vehicle, and using the fused detection results to judge whether a container is single or double, the accuracy of the judgment is improved. Therefore, there is no need to deal with missed or false detection of container holes. The joint judgment method avoids false judgments caused by relying on single information to judge whether a container is single or double.

[0081] 2. By first determining the location of the container area, the image of the box seam area can be more effectively obtained, thereby effectively improving the accuracy of single and double box discrimination.

[0082] 3. According to the position of the center point of the detected rectangular area in the image, the height ratio of the container image cropping is adjusted to ensure that the middle sub-area image is exactly the middle area of ​​the container.

[0083] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0084] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for detecting single and double tanks on a quay crane based on machine vision, characterized in that: include: Acquire multiple container images from a database and annotate each container image to generate a container image set; Establishing a neural network Yolov5 and training the neural network Yolov5 using the container image set to obtain a target detection model; Using the target detection model to perform real-time detection on the container image to be tested to obtain a detection result, wherein the detection result includes: the number of containers on the vehicle, the container holes and the text on the container on the vehicle; and Based on the detection result, it is determined whether the vehicle-mounted container is a single container or a double container. Wherein, obtaining a plurality of container images from a database and annotating each container image to generate a container image set further comprises: obtaining a plurality of historical container images from a database or obtaining a container image to be tested in real time; annotating the containers in each historical container image using a Labelimg tool to establish a first data set; after container detection, randomly cropping highly determined containers in the historical container image or the container image to be tested to generate a historical container sub-image or a container sub-image to be tested; and annotating the container holes and texts in the historical container sub-image using the Labelimg tool to establish a second data set; Among them, randomly cropping the highly determined containers in the historical container image or the container image to be tested to generate the historical container sub-image or the container sub-image to be tested further includes: according to the height and width of the minimum circumscribed rectangle of any two 20-foot containers in the historical container image or the container image to be tested, merging the two minimum circumscribed rectangles of the container on the same container truck into a single minimum circumscribed rectangle; determining the cropping ratio of the container according to the position of the center point of the fused single minimum circumscribed rectangle in the image, and randomly cropping the historical container image or the container image to be tested into an upper sub-region image, a middle sub-region image and a lower sub-region image according to the cropping ratio, wherein the middle sub-region image is subjected to box hole and text detection.

2. The machine vision-based single and double tank detection method for quay cranes according to claim 1 is characterized in that: Judging whether the vehicle-mounted container is a single container or a double container based on the detection result further includes: According to whether the container area in the container image is two 20-foot containers, combined with the container hole and text detection information of the middle sub-area, the total score is calculated, where: When the calculated total score is greater than a threshold, determining that the vehicle-mounted container is a double container; and When the calculated total score is less than or equal to a threshold, the vehicle-mounted container is determined to be a single container.

3. The machine vision-based single and double tank detection method for quay cranes according to claim 1 is characterized in that: The total score is calculated using the following formula: Among them, weight0 represents the weight of the number of containers, whether weight1 has a text weight, and whether weight2 has a box hole weight. R0 represents whether there are two containers, whether R1 has text, and whether R2 has a box hole. If yes, it is 1, and if no, it is 0.

4. The machine vision-based single and double tank detection method for quay cranes according to claim 1 is characterized in that: After determining whether the vehicle-mounted container is a single container or a double container based on the detection result, the method further includes: Transmit the judgment result of single tank or double tank to the quay crane control system; When grabbing the vehicle container on the container truck, the quay crane control system adjusts the spreader form of the quay crane according to whether the vehicle container on the container truck is a single container or a double container, so as to grab a single vehicle container or two vehicle containers.

5. The machine vision-based single and double tank detection method for quay cranes according to claim 1 is characterized in that: Establishing a neural network Yolov5 and training the neural network Yolov5 using the container image set to obtain a target detection model further includes: Establishing a first neural network Yolov5 and training the first neural network Yolov5 using a portion of the first data set to obtain a first target detection model; Performing container detection on another portion of the first data set using the first object detection model to distinguish the height of the container as 20 feet or 40 feet; and After establishing the second data set, a second neural network Yolov5 is established and the second neural network Yolov5 is trained using the second data set to obtain a second target detection model.

6. The machine vision-based single and double tank detection method for quay cranes according to claim 5 is characterized in that: Using the target detection model to perform real-time detection on the container image to be tested to obtain the detection result further includes: Using the first object detection model to perform container detection on the container image to be tested, so as to distinguish the height of the container as 20 feet or non-20 feet; and The container sub-image to be tested is detected using the second object detection model to identify the container holes and the on-vehicle container text in the container sub-image to be tested.

7. The machine vision-based single and double tank detection method for quay cranes according to claim 1 is characterized in that: Acquiring a plurality of container image sets further includes: The first camera and the second camera are installed at opposite sides of the lane on the quay crane beam; Using the first camera and the second camera to capture an image of the container on the container truck; and The captured container images are stored in a database as historical container images and the container images collected in real time are used as the container images to be tested.

8. A single and double tank detection device for a quay crane based on machine vision, characterized in that: include: A container image set generation module, used for acquiring multiple container images from a database and annotating each container image to generate a container image set; A model generation module, used for establishing a neural network Yolov5 and training the neural network Yolov5 using the container image set to obtain a target detection model; A detection module, used to use the target detection model to perform real-time detection on the container image to be tested to obtain a detection result, wherein the detection result includes: the number of containers on the vehicle, the container holes and the text on the container on the vehicle; and A judgment module, used for judging whether the vehicle-mounted container is a single container or a double container based on the detection result; The container image set generation module is further used to: obtain multiple historical container images from a database or obtain container images to be tested in real time; use the Labelimg tool to annotate the containers in each historical container image to establish a first data set; after container detection, randomly crop highly determined containers in the historical container images or the container images to be tested to generate historical container sub-images or container sub-images to be tested; and use the Labelimg tool to annotate the container holes and texts in the historical container sub-images to establish a second data set; Among them, randomly cropping the highly determined containers in the historical container image or the container image to be tested to generate the historical container sub-image or the container sub-image to be tested further includes: according to the height and width of the minimum circumscribed rectangle of any two 20-foot containers in the historical container image or the container image to be tested, merging the two minimum circumscribed rectangles of the container on the same container truck into a single minimum circumscribed rectangle; determining the cropping ratio of the container according to the position of the center point of the fused single minimum circumscribed rectangle in the image, and randomly cropping the historical container image or the container image to be tested into an upper sub-region image, a middle sub-region image and a lower sub-region image according to the cropping ratio, wherein the middle sub-region image is subjected to box hole and text detection.

Citation Information

Patent Citations

  • Container crane sling aligning guide system

    CN103818828A

  • Remote control operation system for rubber-type container gantry crane loading and unloading operation

    CN109368503A

  • Automatic container identifying and positioning method

    CN111243016A

  • Container monitoring method, device and equipment and storage medium

    CN111832345A

  • Bridge crane container truck safety positioning method and system

    CN112528721A