Image recognition obstacle avoidance system for container

Abstract

The invention provides an image recognition obstacle avoidance system for a container. The image recognition obstacle avoidance system comprises a plurality of industrial cameras arranged in an operation area of a container portal crane and an obstacle avoidance control host arranged in a monitoring area, the control host is in communication connection with each industrial camera and the container portal crane, obtains a container hoisting control instruction sent by the container portal crane, and monitors environment information in the operation process of the container portal crane through the industrial cameras; the obstacle avoidance control host acquires an environment image of an operation area of the container portal crane; calling container gantry crane advancing image state information of the environment image of the operation area of the container gantry crane; and according to the type of the bridge obstacle and the distance of the bridge obstacle, judging the bridge interference state of the bridge obstacle to the operation of the container portal crane. The system achieves intelligent obstacle avoidance in time in the operation process of the container portal crane, and provides support for safe operation of the gantry crane.

Description

Technical Field

[0001] The invention belongs to the technical field of container transportation, and in particular relates to an image recognition and obstacle avoidance system for containers. Background Art

[0002] Container terminals are engaged in logistics services such as container loading and unloading, import and export of complete vehicles, loading and unloading of other large goods, transshipment, storage, custody, packing and unpacking, transportation, and warehousing. The production business process adopts the internationally advanced terminal production management system and advanced automated production technology.

[0003] Due to the busy business within the port area, when the container yard crane is in production and operation, the operating trajectories of vehicles, personnel, and large machinery are intertwined. Due to the obstruction of vision and the high driving position, it is difficult for the driver to accurately judge the actual situation at the bottom of the yard crane, and there is a hidden danger of safety accidents. With the development of factory automation, realizing automatic obstacle avoidance of the yard crane is a basic condition for the stable operation of the yard crane. At present, the existing yard cranes are equipped with single-point laser obstacle avoidance to identify obstacles in front of the single-point laser. The obstacle avoidance area is limited, and obstacles in the driving area cannot be accurately detected, and the large machine is linked to form a linkage. Summary of the invention

[0004] The present invention provides an image recognition and obstacle avoidance system for containers. The system can effectively recognize and process bridge obstacles of different types and sizes, thereby improving operation safety and operation efficiency.

[0005] The image recognition obstacle avoidance system for containers includes: a plurality of industrial cameras arranged in the operating area of ​​the container gantry crane and an obstacle avoidance control host arranged in the monitoring area;

[0006] The control host is respectively connected to each industrial camera and the container gantry crane for communication. The control host obtains the container lifting control instructions sent by the container gantry crane and monitors the environmental information during the operation of the container gantry crane through the industrial camera.

[0007] The obstacle avoidance control host obtains the environmental image of the container gantry crane operation area;

[0008] Retrieving the container yard crane travel image status information of the container gantry crane operation area environment image;

[0009] Obtaining bridge obstacle information of the container gantry crane operating area environment image according to the container yard bridge traveling image state information of the container gantry crane operating area environment image based on the container yard bridge obstacle avoidance algorithm;

[0010] The bridge obstacle information of the container gantry crane operation area environment image includes: the bridge obstacle type and bridge obstacle distance in the container gantry crane operation area environment image;

[0011] The interference status of the bridge obstacle on the operation of the container gantry crane is determined according to the type and distance of the bridge obstacle.

[0012] Preferably, the obstacle avoidance control host is used to train the obstacle avoidance algorithm of the container yard crane;

[0013] The training process includes:

[0014] Acquire a first bridge-walk obstacle avoidance image set, wherein the bridge-walk training sample feature set in the first bridge-walk obstacle avoidance image set includes a bridge-walk environment image sample and bridge-walk obstacle information of the bridge-walk environment image sample, wherein the bridge-walk obstacle information includes a bridge-walk obstacle category and a bridge-walk obstacle distance of the bridge-walk environment image sample; the bridge-walk obstacle information of the bridge-walk training sample feature set in the first bridge-walk obstacle avoidance image set includes a plurality of bridge-walk obstacle categories;

[0015] The container yard bridge travel image state information is retrieved for each of the bridge-travel training sample feature sets to obtain the container yard bridge travel image state information of each of the bridge-travel training sample feature sets.

[0016] Preferably, the container yard bridge traveling image state information of the bridge training sample feature set in the obstacle avoidance control host includes bridge environment image sample information of the bridge environment image sample, bridge obstacle category information of the bridge environment image sample and bridge obstacle distance information of the bridge environment image sample.

[0017] Preferably, the obstacle avoidance control host performs a first bridge obstacle avoidance recognition training algorithm of the container yard bridge obstacle avoidance algorithm based on the bridge environment image sample information and the bridge obstacle category information of the bridge environment image sample, wherein the first bridge obstacle avoidance recognition training algorithm is configured to train the container yard bridge obstacle avoidance algorithm to detect the bridge obstacle category in the bridge environment image sample.

[0018] Preferably, a second bridge obstacle avoidance recognition training algorithm of the container yard bridge obstacle avoidance algorithm is performed based on the bridge environment image sample information and the bridge obstacle distance information of the bridge environment image sample, wherein the second bridge obstacle avoidance recognition training algorithm is configured to train the container yard bridge obstacle avoidance algorithm to detect the bridge obstacle distance of the bridge environment image sample.

[0019] Preferably, the obstacle avoidance control host performs the following operations on each bridge-travel training sample feature set: the obstacle avoidance control host extracts bridge-travel environment image sample information of the bridge-travel environment image sample, and the bridge-travel environment image sample information includes the overall color distribution, texture features, and shape features of the image.

[0020] Preferably, the obstacle avoidance control host includes a training set of multiple bridge obstacle images; the obstacle avoidance control host uses the images and the corresponding bridge environment image sample information and bridge obstacle category information to train the container yard bridge obstacle avoidance algorithm.

[0021] Preferably, the obstacle avoidance control host is connected to the container gantry crane through a PLC module to remotely control the operation of the container gantry crane.

[0022] According to another embodiment of the present application, an obstacle avoidance control host is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image recognition obstacle avoidance system for containers is implemented when the processor executes the program.

[0023] According to another embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image recognition and obstacle avoidance system for containers is implemented.

[0024] It can be seen from the above technical solutions that the present invention has the following advantages:

[0025] In the image recognition and obstacle avoidance system for containers provided by this application, by real-time monitoring and identifying the environmental image of the container gantry crane operating area, the system can promptly detect and identify potential bridge obstacles, such as other mechanical equipment, personnel or static obstacles. It can effectively avoid collision accidents caused by human negligence or limited vision, thereby greatly improving the safety of operations. The obstacle avoidance control host can quickly determine the interference status of the bridge obstacle information on the operation of the container gantry crane based on the identified bridge obstacle information, and adjust the lifting control instructions accordingly. It plays a real-time response and decision-making ability, reducing the time required to pause or change the operation plan due to obstacle avoidance.

[0026] The container image recognition and obstacle avoidance system provided by this application uses image recognition technology and obstacle avoidance algorithms to achieve intelligent monitoring of the operating environment of container gantry cranes, reduce the need for manual intervention, and reduce the possibility of human error. In addition, the environmental images and obstacle information collected and processed by the system provide data support for subsequent operation planning, equipment maintenance and safety management.

[0027] The image recognition and obstacle avoidance system for containers provided in this application can adapt to different container yard bridge operating environments and scenarios. By adjusting the obstacle avoidance algorithm and parameters, it can effectively identify and handle bridge obstacles of different types and sizes, improve operation safety, improve operation efficiency, realize intelligent management, and improve the level of automation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0029] Figure 1 This is a schematic diagram of the image recognition obstacle avoidance system for containers;

[0030] Figure 2 This is a scene example of the image recognition obstacle avoidance system for containers;

[0031] Figure 3 This is another on-site example of the image recognition and obstacle avoidance system for containers. DETAILED DESCRIPTION

[0032] In the container image recognition obstacle avoidance system provided below, various embodiments of the present disclosure will be described more fully. The present disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present disclosure to the specific embodiments disclosed herein, but the present disclosure should be understood to cover all adjustments, equivalents and / or alternatives that fall within the spirit and scope of the various embodiments of the present disclosure.

[0033] Hereinafter, the terms "include" or "may include" used in various embodiments of the present disclosure indicate the presence of disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present disclosure, the terms "include", "have", and their cognates are intended only to indicate specific features, numbers, steps, operations, elements, components, or a combination of the foregoing, and should not be understood as first excluding the presence of one or more other features, numbers, steps, operations, elements, components, or a combination of the foregoing or the possibility of adding one or more features, numbers, steps, operations, elements, components, or a combination of the foregoing.

[0034] In various embodiments of the present disclosure, the expression "or" or "at least one of A or / and B" includes any combination or all combinations of the words listed at the same time. For example, the expression "A or B" or "at least one of A or / and B" may include A, may include B, or may include both A and B.

[0035] The expressions (such as "first", "second", etc.) used in the various embodiments of the present disclosure may modify the various constituent elements in the various embodiments, but may not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the elements. The above expressions are only used for the purpose of distinguishing one element from other elements. For example, the first user device and the second user device indicate different user devices, although both are user devices. For example, without departing from the scope of the various embodiments of the present disclosure, the first element may be referred to as the second element, and similarly, the second element may also be referred to as the first element.

[0036] It should be noted that if it is described that one component element is “connected” to another component element, the first component element may be directly connected to the second component element, and a third component element may be “connected” between the first component element and the second component element. Conversely, when one component element is “directly connected” to another component element, it can be understood that there is no third component element between the first component element and the second component element.

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figure 1 The figure is a schematic diagram of an image recognition and obstacle avoidance system for containers in a specific embodiment. The system includes: a plurality of industrial cameras arranged in the operating area of ​​a container gantry crane and an obstacle avoidance control host arranged in the monitoring area.

[0039] The obstacle avoidance control host of this embodiment can be a mobile terminal including a mobile phone, a laptop computer, a personal digital assistant (PDA), a tablet computer (PAD), etc., and a fixed terminal such as a digital TV, a desktop computer, etc. However, it will be understood by those skilled in the art that, in addition to the elements specifically used for mobile purposes, the construction according to the embodiment of the present invention can also be applied to a fixed type of terminal.

[0040] The obstacle avoidance control host may include a processor, a memory, a universal serial bus interface, a wireless communication module, an audio module, a speaker, buttons, a camera, a display screen, etc.

[0041] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the obstacle avoidance control host. In other embodiments of the present application, the obstacle avoidance control host may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0042] The processor may include one or more processing units, for example, the processor may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0043] The processor can be the nerve center and command center of the obstacle avoidance control host. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0044] For the industrial camera of this embodiment, the industrial camera is used to capture static images or videos. For example, the camera 193 can be triggered to start through an application instruction to realize the shooting function, such as shooting and acquiring images of any scene. The industrial camera may include components such as an imaging lens, a filter, and an image sensor. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to converge the light emitted or reflected by all objects in the shooting angle; the filter is mainly used to filter out the redundant light waves in the light; the image sensor can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The image sensor is mainly used to convert the received light signal into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard red green blue (RGB), luminance chrominance (YUV) and other formats.

[0045] In some embodiments, the obstacle avoidance control host obtains the first bridge obstacle avoidance image set, which is a key training data preparation step and lays the foundation for subsequent machine learning model training. In this step, the obstacle avoidance control host collects and organizes a set containing multiple bridge training sample feature sets from various sources. Each bridge training sample feature set may include a previously stored bridge environment image sample, and also includes bridge obstacle information associated with the bridge environment image sample, which may be marked to provide the category of obstacles in the bridge environment image sample and bridge interference status information. The bridge interference status information means that the obstacle has invaded the operating area of ​​the container gantry crane. Of course, some obstacles do not collide with the moving container gantry crane. Some obstacles will collide with the moving container gantry crane, and an alarm prompt is required at this time and processing is required.

[0046] Specifically, the first bridge obstacle avoidance image set may include images of various different bridge obstacles in the operating area of ​​the container gantry crane, as well as the bridge obstacle information corresponding to the images. For example, the container yard bridge travel image status information may include a certain walking device or personnel in the operating area of ​​the container gantry crane, and the corresponding bridge obstacle information will indicate the walking device or personnel, and will also give an assessment of the bridge interference state to determine whether it will cause a collision with the moving container gantry crane. The bridge obstacle information is identified based on the image content using the YOLOv5 algorithm. The algorithm can learn the characteristics of different bridge obstacles.

[0047] In this embodiment, the bridge-travel training sample feature set in the first bridge-travel obstacle avoidance image set is diverse to reflect various scenarios and conditions that may be encountered in actual situations.

[0048] In some embodiments, the obstacle avoidance control host retrieves the container yard bridge travel image status information for each of the bridge-travel training sample feature sets to obtain the container yard bridge travel image status information of each of the bridge-travel training sample feature sets, wherein the container yard bridge travel image status information of the bridge-travel training sample feature set includes the bridge-travel environment image sample information of the bridge-travel environment image sample, the bridge-travel obstacle category information of the bridge-travel environment image sample, and the bridge-travel obstacle distance information of the bridge-travel environment image sample.

[0049] In this embodiment, the image state information of the container yard bridge is retrieved from the bridge-travel training sample feature set, that is, feature extraction. In this process, the obstacle avoidance control host conducts an in-depth analysis of each bridge-travel training sample feature set to extract key information that can represent the image content. This information is called the container yard bridge travel image state information. The retrieval of the container yard bridge travel image state information is crucial because it directly affects the effect and accuracy of the subsequent model training.

[0050] In this embodiment, the obstacle avoidance control host performs the following operations on each bridge-travel training sample feature set: the obstacle avoidance control host extracts bridge-travel environment image sample information of the bridge-travel environment image sample, and the bridge-travel environment image sample information includes the overall color distribution, texture features, shape features, etc. of the image.

[0051] In this embodiment, the obstacle avoidance control host extracts bridge obstacle category information from the bridge environment image sample. This information may involve identifying specific obstacles in the image, that is, various bridge obstacles. By identifying the characteristics of the bridge obstacle, information representing different bridge obstacle categories can be generated.

[0052] In this embodiment, the obstacle avoidance control host extracts the bridge obstacle distance information of the bridge environment image sample. This information is intended to evaluate the degree of bridge interference state of the bridge obstacle in the image to the safety of the container gantry crane operation area. The evaluation of the bridge interference state may be based on factors such as the number, location, running direction of the bridge obstacle and the distance from the container gantry crane operation area.

[0053] In this embodiment, the container yard bridge traveling image state information is usually represented in the form of a mathematical vector, and each vector element represents a specific characteristic value. For example, a container yard bridge traveling image state information vector may contain numerical information about multiple aspects such as color, shape, grayscale, volume, area, etc.

[0054] In this embodiment, key container yard bridge travel image status information is extracted from the bridge training sample feature set. The information will provide a rich data basis for subsequent model training, help the obstacle avoidance control host to more accurately identify bridge obstacles in the operating area of ​​the container gantry crane, and evaluate the interference with the container gantry crane.

[0055] In this embodiment, the obstacle avoidance control host performs a first bridge obstacle avoidance recognition training algorithm of the container yard bridge obstacle avoidance algorithm based on the bridge environment image sample information and the bridge obstacle category information of the bridge environment image sample, wherein the first bridge obstacle avoidance recognition training algorithm is configured to train the container yard bridge obstacle avoidance algorithm to detect the bridge obstacle category in the bridge environment image sample.

[0056] In this embodiment, the container yard bridge obstacle avoidance algorithm can adopt the OpenCV algorithm, the Halcon algorithm, or the YOLOv5 algorithm.

[0057] In this embodiment, the obstacle avoidance control host uses the previously retrieved bridge environment image sample information and bridge obstacle category information to train the first bridge obstacle avoidance recognition training algorithm. The main purpose of the first bridge obstacle avoidance recognition training algorithm is to train the container yard bridge obstacle avoidance algorithm so that it can accurately detect the bridge obstacle category in the image, that is, various bridge obstacles. In order to achieve this goal, the obstacle avoidance control host uses the bridge environment image sample information and bridge obstacle category information as input data to provide the container yard bridge obstacle avoidance algorithm for learning.

[0058] In this embodiment, the bridge environment image sample information may include overall and local features of the image, such as color, shape, grayscale, volume, area and other information, while the bridge obstacle category information focuses on identifying the bridge obstacle features in the image.

[0059] In this embodiment, during the training process of the first bridge obstacle avoidance recognition training algorithm, the obstacle avoidance control host continuously adjusts the parameters of the container yard bridge obstacle avoidance algorithm so that it can better learn and recognize these features. Through a large number of training samples and iterative optimization, the algorithm gradually learns how to accurately extract the features of the bridge obstacle category from the image and can automatically detect these features in new images.

[0060] In this embodiment, the bridge obstacle avoidance recognition training algorithm can use a convolutional neural network. The obstacle avoidance control host continuously adjusts the network weights so that the network output is more consistent with the bridge obstacle category information.

[0061] In this embodiment, the obstacle avoidance control host may include a training set of multiple bridge obstacle images. The obstacle avoidance control host uses the images and the corresponding bridge environment image sample information and bridge obstacle category information to train the container yard bridge obstacle avoidance algorithm. After multiple iterations and optimizations, the algorithm will be able to automatically detect bridge obstacles in new, unseen container gantry crane operation area images.

[0062] In this embodiment, the container yard bridge obstacle avoidance algorithm is trained by utilizing the bridge environment image sample information and the bridge obstacle category information, so that it can accurately detect the bridge obstacle category in the image, providing key technical support for the identification of bridge obstacles in the operating area of ​​the container gantry crane.

[0063] In this embodiment, the second bridge obstacle avoidance recognition training algorithm is trained based on the bridge environment image sample information and the bridge obstacle distance information of the bridge environment image sample, wherein the second bridge obstacle avoidance recognition training algorithm is configured to train the container yard bridge obstacle avoidance algorithm to detect the bridge obstacle distance of the bridge environment image sample.

[0064] In this embodiment, the obstacle avoidance control host uses the bridge environment image sample information and the bridge obstacle distance information to train the second bridge obstacle avoidance recognition training algorithm. Different from the first bridge obstacle avoidance recognition training algorithm, the goal of the second bridge obstacle avoidance recognition training algorithm is to train the container yard bridge obstacle avoidance algorithm so that it can evaluate and detect the bridge interference state of obstacles in the image to the safe operation area of ​​the container gantry crane.

[0065] In this embodiment, the bridge environment image sample information provides overall and local information of the image, and the bridge obstacle distance information includes data on the degree of bridge interference state that may be caused by bridge obstacles to the operation area of ​​the container gantry crane. For example, if an image contains a large number of bridge obstacles close to the operation area of ​​the container gantry crane, then its bridge obstacle distance information will correspondingly reflect a high interference state.

[0066] In this embodiment, during the training process of the second bridge obstacle avoidance recognition training algorithm, the obstacle avoidance control host uses this information to adjust and optimize the parameters of the container yard bridge obstacle avoidance algorithm.

[0067] In this embodiment, the container yard bridge obstacle avoidance algorithm is trained by using the bridge environment image sample information and the bridge obstacle distance information, so that it can accurately evaluate the bridge interference state of obstacles in the image to the safety of the container gantry crane operation area.

[0068] In this embodiment, the obstacle avoidance control host obtains the first bridge-walk obstacle avoidance image set, which may include: obtaining a second bridge-walk obstacle avoidance image set, wherein the second bridge-walk obstacle avoidance image set includes the bridge-walk environment image sample and the bridge-walk obstacle category and bridge-walk obstacle distance of the bridge-walk environment image sample.

[0069] In this embodiment, the bridge walking obstacle category of the bridge walking environment image sample under the bridge walking obstacle category is obtained.

[0070] In this embodiment, the bridge obstacle information of the bridge environment image sample is generated based on the bridge obstacle category of the bridge environment image sample, the bridge obstacle distance of the bridge environment image sample and the bridge obstacle category of the bridge environment image sample under the bridge obstacle category.

[0071] In this embodiment, a bridge-walking training sample feature set is formed according to the bridge-walking environment image samples and the bridge-walking obstacle information, and the first bridge-walking obstacle avoidance image set is formed according to the multiple bridge-walking training sample feature sets.

[0072] In this embodiment, the second bridge travel obstacle avoidance image set is obtained, which may include a plurality of bridge travel environment image samples and the categories of obstacles and bridge travel interference state information in these bridge travel environment image samples.

[0073] In this embodiment, the bridge obstacles in the bridge environment image samples are classified. Specifically, the obstacle avoidance control host further divides the obstacle categories according to the categories of the obstacles and the bridge interference states.

[0074] In this embodiment, the obstacle avoidance control host can use image processing and machine learning technology to automatically complete the classification. The obstacle avoidance control host can locate the bridge obstacles in the image through image segmentation and recognition technology, and then extract the color, shape, grayscale, volume, area and other characteristics of the bridge obstacles. The obstacle avoidance control host can use these features and the pre-set bridge interference state evaluation model to automatically determine the bridge interference state of each bridge obstacle to the safe operation area of ​​the container gantry crane, and classify it into the corresponding category accordingly.

[0075] In some embodiments, it can help the obstacle avoidance control host to more accurately understand the specific interference state of the bridge obstacles in the image on the safety of the operating area of ​​the container gantry crane, thereby ensuring the operating safety of the container gantry crane.

[0076] In this embodiment, the obstacle avoidance control host generates bridge obstacle information based on the bridge obstacle category, bridge obstacle distance and bridge obstacle category in the bridge environment image sample. This not only indicates the type of obstacle in the image, but also includes the degree of bridge interference state that may be caused to the operating area of ​​the container gantry crane.

[0077] In this embodiment, the obstacle avoidance control host forms a bridge walking training sample feature set based on the acquired bridge walking environment image samples and bridge walking obstacle information. The bridge walking training sample feature set is image data with recognition, which can be used as training data for subsequent machine learning processes.

[0078] Specifically, the feature set of bridge operation training samples contains images of the container gantry crane operation area and its surroundings, as well as annotation labels of the categories of bridge operation obstacles, bridge operation interference status, and subdivided categories in these images.

[0079] For example, the bridge traffic training sample feature set includes an image of the operating area of ​​a container gantry crane. There are several bridge traffic obstacles near the operating area of ​​the container gantry crane in the image. At the same time, the template will have corresponding annotation marks, such as bridge traffic obstacles, high bridge traffic interference status, and high bridge traffic interference status bridge traffic obstacles. Such marks clearly indicate the bridge traffic obstacle category, bridge traffic interference status and category in the image.

[0080] This embodiment constructs a sufficient number of bridge-traffic training sample feature sets, and the obstacle avoidance control host will combine these templates to form a first bridge-traffic obstacle avoidance image set. The set is the basic data set for training the container yard bridge obstacle avoidance algorithm. It contains rich image features and corresponding identification information, which can accurately identify the obstacle type from the image and evaluate the ability of the bridge-traffic obstacle to the interference state of the container gantry crane in the operation area.

[0081] In some specific embodiments, in order to solve the obstacle avoidance safety problem of container yard cranes, the currently advanced mainstream image recognition obstacle avoidance method is adopted. Existing and installed cameras are planned to be used before and after the container yard crane, facing the travel direction of the track respectively, to obtain the actual scene pictures on site. The recognition algorithm adopts a deep convolutional neural network, and the system architecture adopts an edge computing architecture. The images collected by the camera are transmitted back to the edge computing host. Through the independently developed target recognition algorithm, the computing host analyzes and identifies the target vehicles, personnel, large machinery, etc. in real time. When the target object enters the dangerous area, the edge computing host automatically feeds back to the control host PLC of the container yard crane through the serial port signal, controls the container yard crane to make intelligent risk avoidance in time, and provides basic data for the obstacle avoidance strategy of the yard crane operation.

[0082] In some embodiments, Figure 2 and Figure 3As shown in the figure, in order to solve the obstacle avoidance safety problem, the currently advanced mainstream image recognition obstacle avoidance method is adopted. The existing and installed cameras are used to face the direction of the track to obtain the actual scene pictures on site. The recognition algorithm adopts a deep convolutional neural network, and the obstacle avoidance control host adopts an edge computing architecture. The images collected by the camera are transmitted back to the obstacle avoidance control host. Through the independently developed target recognition algorithm, the computing host analyzes and identifies the target vehicles, personnel, large machinery, etc. in real time. When the target enters the dangerous area, it is automatically fed back to the obstacle avoidance control host through the serial port signal, and the control makes intelligent risk avoidance in time, providing basic data for the operation and obstacle avoidance strategy of the field crane.

[0083] For example, through the visual target ranging algorithm, people, vehicles, large machinery and other equipment in the target area can be automatically and accurately identified; people, vehicles, and large equipment at long (13m) and short (3m) distances can be identified, and different serial port signals at different distances can be obtained through edge computing; different serial port signal information can be communicated with the control host PLC of the container yard crane to control the container yard crane to make corresponding risk avoidance actions (slow down or stop) in time.

[0084] Motion control conditions

[0085]

[0086] The container image recognition and obstacle avoidance system involved in this embodiment uses visual analysis and deep learning algorithms to identify obstacles such as field cranes, container transport vehicles, engineering vehicles, personnel, and bags that are likely to appear in the field crane driving area. It realizes real-time measurement of the distance to the target obstacle and the distance to the target object. By converting different recognition distances into control serial port signal information and transmitting it to the obstacle avoidance control host, the obstacle avoidance control host can make corresponding risk avoidance actions in time, such as controlling the container gantry crane to slow down or stop.

[0087] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0088] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the equipment, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. Exemplarily, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0089] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0090] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0091] In the storage medium provided in the present application, a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0092] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image recognition obstacle avoidance system for containers, characterized in that: include: Multiple industrial cameras installed in the operating area of ​​the container gantry crane and an obstacle avoidance control host installed in the monitoring area; The control host is respectively connected to each industrial camera and the container gantry crane for communication. The control host obtains the container lifting control instructions sent by the container gantry crane and monitors the environmental information during the operation of the container gantry crane through the industrial camera. The obstacle avoidance control host obtains the environmental image of the container gantry crane operation area; Retrieving the container yard crane travel image status information of the container gantry crane operation area environment image; Obtaining bridge obstacle information of the container gantry crane operating area environment image according to the container yard bridge traveling image state information of the container gantry crane operating area environment image based on the container yard bridge obstacle avoidance algorithm; The bridge obstacle information of the container gantry crane operation area environment image includes: the bridge obstacle type and bridge obstacle distance in the container gantry crane operation area environment image; The interference status of the bridge obstacle on the operation of the container gantry crane is determined according to the type and distance of the bridge obstacle.

2. The image recognition obstacle avoidance system for containers according to claim 1, characterized in that: The obstacle avoidance control host is used to train the obstacle avoidance algorithm for container yard cranes; The training process includes: Acquire a first bridge-walk obstacle avoidance image set, wherein the bridge-walk training sample feature set in the first bridge-walk obstacle avoidance image set includes a bridge-walk environment image sample and bridge-walk obstacle information of the bridge-walk environment image sample, wherein the bridge-walk obstacle information includes a bridge-walk obstacle category and a bridge-walk obstacle distance of the bridge-walk environment image sample; the bridge-walk obstacle information of the bridge-walk training sample feature set in the first bridge-walk obstacle avoidance image set includes a plurality of bridge-walk obstacle categories; The container yard bridge travel image state information is retrieved for each of the bridge-travel training sample feature sets to obtain the container yard bridge travel image state information of each of the bridge-travel training sample feature sets.

3. The image recognition obstacle avoidance system for containers according to claim 2, characterized in that: The container yard bridge traveling image state information of the bridge training sample feature set in the obstacle avoidance control host includes bridge environment image sample information of the bridge environment image sample, bridge obstacle category information of the bridge environment image sample and bridge obstacle distance information of the bridge environment image sample.

4. The image recognition obstacle avoidance system for containers according to claim 3, characterized in that: The obstacle avoidance control host performs a first bridge obstacle avoidance recognition training algorithm of the container yard bridge obstacle avoidance algorithm based on the bridge environment image sample information and the bridge obstacle category information of the bridge environment image sample, wherein the first bridge obstacle avoidance recognition training algorithm is configured to train the container yard bridge obstacle avoidance algorithm to detect the bridge obstacle category in the bridge environment image sample.

5. The image recognition obstacle avoidance system for containers according to claim 4, characterized in that: According to the bridge environment image sample information and the bridge obstacle distance information of the bridge environment image sample, a second bridge obstacle avoidance recognition training algorithm of the container yard bridge obstacle avoidance algorithm is performed, wherein the second bridge obstacle avoidance recognition training algorithm is configured to train the container yard bridge obstacle avoidance algorithm to detect the bridge obstacle distance of the bridge environment image sample.

6. The image recognition obstacle avoidance system for containers according to claim 1 or 2, characterized in that: The obstacle avoidance control host performs the following operations on each bridge training sample feature set: the obstacle avoidance control host extracts bridge environment image sample information of the bridge environment image sample, and the bridge environment image sample information includes the overall color distribution, texture features, and shape features of the image.

7. The image recognition obstacle avoidance system for containers according to claim 1 or 2, characterized in that: The obstacle avoidance control host contains a training set of various bridge obstacle images; the obstacle avoidance control host uses the images and the corresponding bridge environment image sample information and bridge obstacle category information to train the container yard bridge obstacle avoidance algorithm.

8. The image recognition obstacle avoidance system for containers according to claim 1 or 2, characterized in that: The obstacle avoidance control host is connected to the container gantry crane through the PLC module to remotely control the operation of the container gantry crane.

9. An obstacle avoidance control host, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image recognition and obstacle avoidance system for containers as described in any one of claims 1 to 8 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image recognition and obstacle avoidance system for containers as claimed in any one of claims 1 to 8 is implemented.

Images