Obstacle avoidance system and method based on computer vision

Abstract

The invention relates to the technical field of security and protection monitoring of port operation areas, in particular to an obstacle avoidance system and method based on computer vision. An obstacle avoidance system based on computer vision comprises an image acquisition module used for acquiring port image data; the target mechanical equipment detection module is integrated with a first visual detection model and is used for performing reasoning detection on target mechanical equipment image data in the mechanical equipment operation area to obtain a first detection result; the operator detection module is integrated with a second visual detection model and is used for carrying out parallel reasoning detection on the image data of the operation area of the operator to obtain a second detection result; and the risk assessment module is used for judging whether the target mechanical equipment or the operating personnel invade a risk area or not based on the first detection result and the second detection result. According to the invention, self-adaptive collision avoidance / obstacle avoidance is realized, and dynamic obstacle avoidance and alarm response of mechanical equipment and operators in the operation process are ensured.

Description

Technical Field

[0001] This invention relates to the field of security monitoring technology in port operation areas, and in particular to an obstacle avoidance system and method based on computer vision. Background Technology

[0002] In bulk cargo terminal loading and unloading operations, grab cranes (gantry cranes) lift bulk cargo from the ship's hold to the shore. During this process, the grab crane must work in coordination with mobile machinery (such as excavators and bulldozers, collectively referred to as "mobile machinery") and personnel within the confined space of the ship's hold. Due to the complex structure inside the hold, severe obstruction of vision, and the dynamic and ever-changing working environment, there is an extremely high risk of collision between the grab crane, mobile machinery, and personnel. Traditional monitoring methods relying on operator visual observation are limited by human attention, reaction speed, and blind spots, making it difficult to achieve all-weather, comprehensive safety monitoring. This can easily lead to response delays and misjudgments, posing a significant safety hazard to production.

[0003] To enhance safety, current technologies typically install downward-looking cameras on the boom of the gantry crane to capture panoramic video of the cabin. However, current video-based monitoring solutions suffer from several limitations: First, most systems still rely on manual observation of video footage for judgment, resulting in low automation and insufficient efficiency and real-time performance to meet the demands of high-speed operations. Second, some automated video analysis systems cannot accurately perceive the real-time attitude and orientation of targets with rotational characteristics. Since the collision hazard zone of the grab is closely related to its attitude, the horizontal frame contains a large amount of irrelevant background and cannot accurately reflect the grab's outline and rotation angle, preventing the system from dynamically and accurately constructing the surrounding hazard zone (warning zone and alarm zone) based on the grab's true attitude. Therefore, existing technologies struggle to achieve an automated collision avoidance solution that can adapt to target attitude, perform accurate real-time risk assessment, and integrate with industrial control systems.

[0004] In summary, there is an urgent need for an intelligent safety system that can accurately sense the attitude of rotating targets, dynamically assess collision risks, and achieve automatic and real-time linkage in order to solve the core safety problems faced in bulk cargo loading and unloading operations. Summary of the Invention

[0005] The following provides a brief overview of one or more aspects to offer a basic understanding of them. This overview is not an exhaustive summary of all conceived aspects, nor is it intended to identify key or decisive elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form to prepare for the more detailed descriptions that follow.

[0006] The purpose of this invention is to provide a computer vision-based obstacle avoidance system that solves the technical problem of human-machine collision avoidance in bulk cargo loading and unloading at ports.

[0007] To achieve the above objectives, the present invention provides a computer vision-based obstacle avoidance system, the system comprising at least: an image acquisition module, a target mechanical equipment detection module, a worker detection module, and a risk assessment module.

[0008] The image acquisition module is used to acquire port image data, which includes image data of mechanical equipment and the work area of ​​operators;

[0009] The target mechanical equipment detection module integrates a first visual detection model, and performs inference detection on the target mechanical equipment image data in the mechanical equipment operation area based on the first visual detection model to obtain a first detection result.

[0010] The worker detection module integrates a second visual detection model, and performs parallel inference detection on the image data of the worker's work area based on the second visual detection model to obtain a second detection result; and

[0011] The risk assessment module is communicatively connected to the target machinery and equipment detection module and the operator detection module. Based on the first detection result and the second detection result, it determines whether the target machinery and equipment or the operator has entered the risk area, so as to generate a corresponding early warning or alarm signal.

[0012] In one embodiment, the first visual detection model detects the target mechanical equipment image data using a rotated rectangle and outputs a first detection result;

[0013] The second visual detection model uses a horizontal rectangular bounding box to detect the image data of the worker's work area and outputs a second detection result.

[0014] In one embodiment, the first visual detection model is trained based on a dataset of target mechanical equipment containing rotated bounding box annotations;

[0015] The second visual detection model was trained on a people dataset containing horizontal rectangular labels.

[0016] In one embodiment, the risk assessment module generates at least one concentric magnified region as a risk region based on the rotating rectangular frame of the mechanical equipment;

[0017] The concentric magnification region includes a first magnification region and a second magnification region that are concentric with the rotating rectangular frame of the mechanical equipment, wherein the magnification factor of the first magnification region is less than that of the second magnification region.

[0018] In one embodiment, a target stabilization module is also included:

[0019] The target stabilization module is communicatively connected to the mechanical equipment target detection module and the operator detection module, and uses a multi-frame stabilization algorithm to perform time-series stabilization processing on the first detection result and the second detection result.

[0020] In one embodiment, the target stabilization module performs time-series stabilization processing on the first detection result and the second detection result, further including:

[0021] Based on the set minimum number of active frames, a smoothing algorithm is used to filter the first and second detection results;

[0022] By calculating the state changes of the target between consecutive frames, mutation detection results exceeding a preset threshold are filtered out;

[0023] Output the stabilized target information.

[0024] In one embodiment, the image acquisition module further includes: analyzing the quality of the image data to detect whether the image acquisition module has malfunctioned;

[0025] In response to the detection of a malfunction in the image acquisition module, the image processing parameters are adjusted or the system switches to a backup image acquisition module.

[0026] In one embodiment, an industrial communication module is also included:

[0027] The industrial communication module is communicatively connected to the risk assessment module and is used to communicate with the programmable logic controller of the port equipment via the TCP protocol to transmit the early warning or alarm signal to the programmable logic controller.

[0028] In one embodiment, the target mechanical device is a mobile mechanical device.

[0029] To achieve the above objectives, the present invention provides a method, the method being operated on any of the systems described above, the method comprising:

[0030] Acquire port image data, including image data of machinery and equipment and the work areas of workers;

[0031] Inference is performed on the image data of the target mechanical equipment in the mechanical equipment operation area;

[0032] Parallel inference is performed on the image data of the worker's work area;

[0033] Determine whether the target machinery or personnel have intruded into the risk area, and generate a corresponding warning or alarm signal.

[0034] This invention provides a computer vision-based obstacle avoidance system and method. By designing and training dedicated visual models for mechanical equipment and personnel respectively, it can optimize for the shape, scale and motion characteristics of different targets, and integrate an automated risk assessment module to achieve adaptive collision avoidance / obstacle avoidance. This significantly reduces the collision risk in port operations and ensures dynamic obstacle avoidance and alarm response for mechanical equipment and personnel during operations. Attached Figure Description

[0035] The above-described features and advantages of the present invention will be better understood after reading the following detailed description of embodiments of the present disclosure in conjunction with the accompanying drawings. In the drawings, components are not necessarily drawn to scale, and components having similar related characteristics or features may have the same or similar reference numerals.

[0036] Figure 1 A block diagram of a computer vision-based obstacle avoidance system according to an embodiment of the present invention is disclosed;

[0037] Figure 2 A schematic diagram of a user interface according to an embodiment of the present invention is shown;

[0038] Figure 3 A schematic diagram of a camera configuration according to an embodiment of the present invention is shown;

[0039] Figure 4 A schematic diagram of a window for setting the Modbus TCP communication address according to an embodiment of the present invention is shown;

[0040] Figure 5 A schematic diagram of the first interface according to an embodiment of the present invention is disclosed;

[0041] Figure 6 A schematic diagram of the second interface according to an embodiment of the present invention is disclosed;

[0042] Figure 7 The diagram illustrates the steps of an obstacle avoidance method based on computer vision according to an embodiment of the present invention.

[0043] Figure 8 A flowchart of an obstacle avoidance method based on computer vision according to an embodiment of the present invention is disclosed. Detailed Implementation

[0044] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention will be presented in conjunction with preferred embodiments, this does not mean that the features of the invention are limited to these embodiments. On the contrary, the purpose of describing the invention in conjunction with embodiments is to cover other options or modifications that may be derived based on the claims of the present invention. To provide a deep understanding of the invention, many specific details will be included in the following description, and the invention may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of the invention, some specific details will be omitted in the description.

[0045] Figure 1 The diagram illustrates a computer vision-based obstacle avoidance system according to an embodiment of the present invention, such as... Figure 1 As shown, a computer vision-based obstacle avoidance system includes at least an image acquisition module 100, a target machinery detection module 201, a worker detection module 202, and a risk assessment module 300.

[0046] The image acquisition module 100 is used to acquire port image data, which includes image data of mechanical equipment and the work area of ​​operators;

[0047] The target mechanical equipment detection module 201 integrates a first visual detection model, and performs inference detection on the target mechanical equipment image data in the mechanical equipment operation area based on the first visual detection model to obtain a first detection result.

[0048] The worker detection module 202 integrates a second visual detection model, and performs parallel inference detection on the image data of the worker's work area based on the second visual detection model to obtain a second detection result; and

[0049] The risk assessment module 300 is communicatively connected to the target machinery and equipment detection module 201 and the operator detection module 202. Based on the first detection result and the second detection result, it is used to determine whether the target machinery and equipment or the operator has entered the risk area, so as to generate a corresponding early warning or alarm signal.

[0050] This invention proposes a computer vision-based obstacle avoidance system. By designing and training dedicated visual models for both machinery and personnel, it can optimize for the shape, scale, and motion characteristics of different targets and integrate an automated risk assessment module to achieve adaptive collision avoidance / obstacle avoidance. It can identify the grab's posture and its relative relationship with surrounding targets (machines, personnel, etc.) in real time and with high accuracy, dynamically construct warning / alarm zones, and output control signals in conjunction with a programmable logic controller (PLC) through an industrial communication protocol. This significantly reduces the collision risk in port operations and ensures dynamic obstacle avoidance and alarm response for machinery and personnel during operations.

[0051] The following is combined with Figure 1 The computer vision-based obstacle avoidance system provided by the present invention will be described in detail with specific embodiments.

[0052] The image acquisition module 100 is used to acquire port image data, which includes image data of mechanical equipment and the work area of ​​the operators.

[0053] The image acquisition module 100 can be a camera. Preferably, the camera is fixedly installed as a top-down camera on the gantry crane's trunk beam (directly above the grab bucket) to acquire real-time video streams of the work area. To build and optimize the model, approximately 16 hours of actual machine operation video recordings covering various complex scenarios such as daytime, nighttime, and heavy rain were first collected to ensure the diversity and robustness of the image data and improve the accuracy of target detection.

[0054] In one embodiment, based on image quality analysis of real-time acquired images, it is detected whether the image acquisition module 100 has occlusion, rain effects, or overexposure faults.

[0055] In response to a detected fault, the system adaptively adjusts image processing parameters or switches to a backup image acquisition module to maintain the continuous operation of the monitoring system.

[0056] Specifically, the system detects whether an image is occluded by calculating its brightness distribution and standard deviation. If the standard deviation is less than a threshold, the image is considered occluded. Edge detection and blur analysis determine whether the image is affected by rain. Numerous blurred areas or excessive edge noise can be attributed to rain. Finally, the system analyzes the image's mean brightness; if the mean is too high, the image is considered overexposed.

[0057] To ensure the system's reliability in complex environments, fault detection is performed on the image acquisition module, and the target detection strategy is automatically adjusted, such as dynamically reducing image resolution, adjusting the processing frame rate, or switching to a backup camera, thereby ensuring the continuous and stable operation of the monitoring function.

[0058] In practical implementation, the image data acquired by the image acquisition module 100 should be interpreted broadly. It not only refers to static image data captured at a single point in time, but more importantly, it encompasses continuous, time-series dynamic image data composed of video streams. This video sequence-based image data forms the basis for the system to achieve high-precision target detection and tracking, and to perform displacement calculations.

[0059] In one embodiment, the obstacle avoidance system further includes a data processing module, which receives image data acquired by the image acquisition module 100 and preprocesses the raw video data, including extracting image frames at fixed frame intervals and cropping and enhancing the extracted image frames to obtain a basic image set.

[0060] In this embodiment, the video data is first sliced. During the real-time operation phase, the acquired video stream is split into continuous image frames to serve as the basis for subsequent model training.

[0061] Specifically, images are extracted at fixed intervals (e.g., every 50 frames) and cropped or scaled to a uniform base resolution (e.g., 1280×800). Alternatively, image processing software (e.g., OpenCV) can be used to process the images, such as adjusting brightness and contrast, and performing color space conversion, to ensure that the image quality is suitable for training the object detection model and forming the original image dataset.

[0062] The target mechanical equipment detection module 201 integrates a first visual detection model. Based on the first visual detection model, it performs inference detection on the target mechanical equipment image data in the mechanical equipment operation area to obtain a first detection result.

[0063] In one embodiment, the first visual detection model detects the image data of the target mechanical equipment using a rotating rectangular bounding box and outputs a first detection result. The target mechanical equipment is a mobile mechanical device, which may include a grab bucket, the front and rear halves of a bulldozer, and the boom and body of an excavator.

[0064] The first detection result is the position and attitude angle of the target mechanical equipment.

[0065] The first-line vision detection model is used to detect grabs and flow machines. It is a model that is detected and trained using rotating rectangular boxes (which can predict the orientation of the equipment).

[0066] Among them, the Oriented Bounding Box (OBB) is a type of bounding box used for object detection, which can more accurately describe the boundaries of rotated or tilted objects.

[0067] In one embodiment, the first visual detection model is trained on a dataset of target mechanical equipment containing rotating box annotations.

[0068] In this embodiment, in order to train a high-precision visual detection model, the preprocessed original image dataset is finely annotated for different targets (mechanical equipment and personnel), and a target mechanical equipment image dataset is constructed. The dataset includes at least a training set, a validation set, and a test set. The training set is used to train the target detection model, the validation set is used to validate the model, and the test set is used to test the performance of the model.

[0069] The target machinery image dataset focuses on machinery such as grab buckets, excavators, and bulldozers, using rotated bounding boxes for annotation. To more accurately represent the equipment outlines and reduce background interference, component-level annotations were performed on some complex machines. For example, excavators are labeled as "digging arm" and "body," and bulldozers are labeled as "front half" and "rear half." This dataset is used to train a first-person vision detection model (machine target detection model).

[0070] Based on the aforementioned target mechanical equipment image dataset, a first visual detection model was trained:

[0071] The input resolution was set to 640×640, and the model was trained using a rotated target detection algorithm. The model outputs the bounding box of the target machinery, i.e., a rotated rectangle, which contains the target's center coordinates, length, width, and, crucially, rotation angle.

[0072] By using a rotating rectangle to mark and record the angle information of target mechanical equipment (such as grabs in bulk cargo equipment and flow machines in ship holds and on shore), the spatial attitude of target mechanical equipment can be accurately represented, adapting to various complex operating scenarios.

[0073] The worker detection module 202 integrates a second visual detection model. Based on the second visual detection model, it performs parallel inference detection on the image data of the worker's work area to obtain a second detection result.

[0074] In one embodiment, the second visual detection model detects the image data of the worker's work area using a horizontal rectangular bounding box and outputs a second detection result.

[0075] The second detection result is the target location of the operator.

[0076] The second visual detection model is a model for labeling small target people, which is a model that uses horizontal rectangular boxes for detection and training.

[0077] Among them, the Horizontal Bounding Box (HBB) is a labeling method that emphasizes enclosing an object in the horizontal direction, and is suitable for situations where the object is roughly horizontal or vertical.

[0078] In one embodiment, the second visual detection model is trained on a people dataset containing horizontal rectangular labels.

[0079] In this embodiment, in order to train a high-precision visual detection model, the preprocessed original image dataset is finely annotated for the workers, and a target worker dataset is constructed.

[0080] The target workforce dataset is labeled with horizontal rectangular boxes to further ensure that the model can identify people of different sizes and positions. Since people are usually small targets in an overhead view, this improves the recall and precision for small targets.

[0081] Based on the aforementioned target worker dataset, a second visual detection model was trained:

[0082] The input resolution is set to 960×960. The system detects workers and outputs their bounding boxes. A horizontal target detection algorithm is used for training, and data augmentation techniques (such as image scaling and random rotation) are employed to improve the model's robustness to small targets. Alternatively, a higher input resolution can be used, which helps the network extract more refined features, thereby accurately detecting smaller pedestrian targets in the image.

[0083] The second video detection model is used to identify workers and can effectively detect people moving around the machinery. Even small, distant targets can be accurately identified.

[0084] In actual deployment, the first visual detection model and the second visual detection model are loaded into the corresponding detection modules and perform parallel inference on the same frame image from the image acquisition module 100, respectively outputting the detection results of mechanical and personnel targets.

[0085] During training, cross-validation and early stopping techniques are employed to avoid overfitting, and the Adam optimizer and multi-scale training are used to improve the model's accuracy and generalization ability. Adam (Adaptive Moment Estimation) is a widely used deep learning optimization algorithm that adjusts the learning rate of each parameter by calculating the first and second moment estimates of the gradient, thereby achieving more efficient network training.

[0086] The use of a single horizontal bounding box detection model to simultaneously identify grabs, mobile machinery, and personnel resulted in excessive background interference in the horizontal bounding boxes for mechanical equipment, making it impossible to accurately delineate their outlines and leading to inaccurate spatial position and orientation determination. Subsequently, rotating bounding boxes were used for annotation and a dedicated model was trained to accurately capture the orientation of the mechanical equipment, improving the detection of small, distant personnel. However, increasing the model's input resolution led to a decrease in the detection accuracy and stability of large-sized mobile machinery; conversely, using a lower resolution made personnel targets easily confused with similar-shaped and scaled objects in the scene.

[0087] This embodiment employs a dual-model parallel inference architecture, including a rotating target detection model for mechanical equipment and a horizontal target detection model optimized for small targets such as people. It achieves high-precision perception of the mechanical equipment's posture and high-recall recognition of human targets. By synchronously analyzing the same frame of image, it realizes balanced and accurate detection of targets with different characteristics in complex scenes.

[0088] In one embodiment, the system further includes a target stabilization module:

[0089] The target stabilization module is communicatively connected to the mechanical equipment target detection module and the operator detection module, receives the first detection result and the second detection result, and uses a multi-frame stabilization algorithm to perform time-series stabilization processing on the first detection result and the second detection result.

[0090] In one embodiment, the time-series stabilization processing of the first detection result and the second detection result further includes:

[0091] Based on the set minimum number of active frames, a smoothing algorithm is used to filter the first and second detection results;

[0092] By calculating the state changes of the target between consecutive frames, mutation detection results exceeding a preset threshold are filtered out;

[0093] Output the stabilized target information.

[0094] In this embodiment, the original detection results are susceptible to changes in lighting, camera shake, and fluctuations in the model itself, resulting in target flickering or sudden changes in position / angle. The target stabilization module receives the original results from the target machinery detection module and the worker detection module, and implements a multi-frame stabilization algorithm.

[0095] Specifically, by setting a minimum number of active frames and a position smoothing algorithm, false detections and flickering caused by factors such as camera shake and changes in lighting are reduced, short-term shake and false detections are suppressed, and target detection is made more stable and reliable.

[0096] The system calculates the target's center displacement, area change, and angle change between adjacent frames. If the change exceeds a preset physically reasonable threshold, it is identified as a sudden change and is filtered out or smoothed.

[0097] The location, size, and confidence level of the filtered targets are smoothed over time (e.g., using Kalman filtering or moving average), and the stabilized target information is output.

[0098] In this embodiment, the detection results between consecutive frames are first matched and tracked for the target range region, and valid targets are confirmed by counting the appearance and disappearance of targets across multiple frames. Based on this, abrupt changes in key parameters such as the target's center displacement, area, and angle are filtered to eliminate instantaneous jumps that do not conform to physical laws (especially for the angle data of the grab target, which can be set in the .json file). Finally, the position and confidence scores of the filtered targets are temporally smoothed to ensure that only target information that has reached a stable state is output, thereby effectively suppressing false detections, jitter, and flickering, and improving detection reliability and stability.

[0099] The risk assessment module 300 is communicatively connected to the target machinery and equipment detection module 201 and the operator detection module 202. Based on the first detection result and the second detection result, it is used to determine whether the target machinery and equipment or the operator has entered the risk area, so as to generate a corresponding early warning or alarm signal.

[0100] Based on the position, angle, and speed of the target machinery or personnel, the risk zone between targets is calculated. For machinery and personnel, the safe distance between them is calculated, and a concentric expansion frame is generated. Then, based on factors such as the degree of overlap, angle changes, and distance between targets, it is determined whether a warning or alarm signal needs to be issued. This is then linked with the industrial control system via the Modbus TCP interface to achieve a closed loop.

[0101] In one embodiment, the risk assessment module generates at least one concentric magnified region as a risk region based on the rotating rectangular frame of the mechanical equipment;

[0102] The concentric magnification region includes a first magnification region and a second magnification region that are concentric with the rotating rectangular frame of the mechanical equipment, wherein the magnification factor of the first magnification region is less than that of the second magnification region.

[0103] The first magnified area is the alarm area; the second magnified area is the warning area.

[0104] In this embodiment, risk areas are dynamically defined through a configuration file. The construction rules are based on the detected grab bucket rotating bounding box (OBB). Specifically, two levels of safety thresholds—warning and alarm—are preset. These thresholds are based on the size of the grab bucket OBB frame, and corresponding risk areas are formed by concentric parallel scaling. For example, the warning area can be set as an N-fold magnified area of ​​the grab bucket OBB frame, and the alarm area can be set as an M-fold magnified area of ​​the grab bucket OBB frame (where N>M). In real-time monitoring, when other targets (such as machines or personnel) enter the warning area, the system triggers a warning signal; if they intrude into a more internal alarm area, a higher-priority alarm signal is triggered, thus achieving graded and adaptive collision risk warning.

[0105] In one embodiment, the system further includes an industrial communication module:

[0106] The industrial communication module is communicatively connected to the risk assessment module and is used to communicate with the programmable logic controller of the port equipment via the TCP protocol to transmit the early warning or alarm signal to the programmable logic controller.

[0107] In this embodiment, the Modbus TCP protocol is selected as the core communication interface to achieve real-time data exchange with devices such as programmable logic controllers (PLCs). After receiving the detection command from the industrial control system (PLC), the system writes the target's position, attitude angle information, and triggered warning and alarm signals into a predefined register address in real time, completing the closed-loop transmission from detection results to control signals. To further ensure the communication reliability in the industrial environment, a timeout and automatic retry mechanism is also set up to maintain or restore the connection in the event of network fluctuations or interruptions, ensuring the continuous and stable operation of the safety monitoring link.

[0108] The computer vision-based obstacle avoidance system provided by this invention can be deployed on edge computing platforms and utilizes GPU acceleration for inference optimization of the visual detection model, significantly reducing single-frame processing time. By constructing a low-latency data processing pipeline, the system ensures that the entire process from image acquisition, target detection, stabilization processing, risk assessment to alarm signal generation can be completed in a very short time, meeting the stringent timeliness requirements of high-load port operations. Furthermore, the system adopts a highly modular design, with each core functional module capable of independent deployment, configuration, and upgrades, thus possessing excellent system integration and functional scalability. It can flexibly adapt to existing monitoring systems in different ports and interact with various external systems.

[0109] The above operation process will be explained in detail below through a specific embodiment.

[0110] Figure 2 A schematic diagram of a user interface according to an embodiment of the present invention is shown, such as... Figure 2 As shown, the user interface was developed using PyQt5, a Python library for creating graphical user interfaces (GUIs). The user interface can include camera configuration, communication configuration, video inspection devices, and start / stop detection.

[0111] Figure 3 A schematic diagram of a camera configuration according to an embodiment of the present invention is shown, such as... Figure 3 As shown, the camera configuration information includes: camera IP address, camera brand, camera login username, and camera login password, etc.

[0112] Figure 4 A schematic diagram of a window for setting the Modbus TCP communication address according to an embodiment of the present invention is shown, as follows: Figure 4 As shown, this window is used to configure Modbus TCP communication parameters with industrial control systems (such as PLCs):

[0113] Server IP (ModbusTCP server IP): In this example, it is 127.0.0.1;

[0114] Port: Modbus TCP default port 502 is used;

[0115] After configuration, click "Confirm". The parameters will be saved to a JSON format config file, thus achieving persistence of the communication configuration.

[0116] Setting the Modbus TCP communication address is crucial for data interaction between the computer vision-based obstacle avoidance system and the PLC, ensuring that detection signals and alarm information can be accurately transmitted between devices.

[0117] Next, configure the video address by entering the complete video file path. This will be used to detect the acquired grab bucket machine video data and verify the model.

[0118] The obstacle avoidance system also supports switching between detection modes, including video file detection and real-time camera stream detection. When "Video Processing" is selected, the system will detect video files based on the configured video file addresses. When "Cam Processing" is selected, it will directly call the configured IP cameras for real-time detection. Typically, Video Processing is used for model verification, while Cam Processing is used for real-time deployment.

[0119] The obstacle avoidance system supports connection status display. You can set the ModbusTCP connection and disconnection checkboxes. When the checkbox is selected and you successfully connect to the ModbusTCP server, the word "Connected" in bold green will be displayed next to it; if you fail, the word "Disconnected" will be grayed out.

[0120] During the detection process, the interface dynamically and visually displays the detection results: the grab bucket is marked with a blue rotating rectangle and its real-time angle is displayed, and multiple concentric risk zones are generated based on this frame; when other targets intrude into the risk zone, the interface provides warnings / alarms through highlighting and color changes, and simultaneously identifies the detected personnel with a green horizontal rectangle.

[0121] Figure 5 A schematic diagram of the first interface according to an embodiment of the present invention is shown, as follows: Figure 5 As shown, during the video processing, the video is being detected. When the grab is detected, it's indicated by a blue rotating rectangle. Two magnified areas are set around the perimeter. When a stream enters the first frame, the area between the outermost and innermost frames becomes solid, and the "warning" text turns orange and bold. Simultaneously, if Modbus TCP is active and the PLC issues a detection command, the warning is sent back to the PLC via Modbus TCP. The angle display also shows that the actual angle of the grab on the screen is currently 1.8°.

[0122] Figure 6 A schematic diagram of the second interface according to an embodiment of the present invention is shown, as follows: Figure 6 As shown, by configuring and adding a second visual detection model for pedestrian detection, the enabled setting for the second model in the configuration file is changed to true. Furthermore, the model inference parameters are set according to requirements. It can be clearly seen that when the second model is enabled, the system will simultaneously detect small pedestrian targets using the HBB method and incorporate them into the logical processing. The green box indicates a person.

[0123] When the Cam Processing function is activated, the real-time angle can be displayed.

[0124] IP cameras achieve low-latency video streaming by calling the webRTC library of aiortc. The time from image acquisition to interface display can be controlled within 200 milliseconds, ensuring real-time monitoring and interaction.

[0125] Figure 7 The diagram illustrates the steps of a computer vision-based obstacle avoidance method according to an embodiment of the present invention, as follows: Figure 7As shown, based on the above-described computer vision-based obstacle avoidance system, this invention proposes a computer vision-based obstacle avoidance method. The method operates on the system described above and includes the following steps:

[0126] S1. Acquire port image data, the image data including image data of mechanical equipment and the work area of ​​operators;

[0127] S2. Infer the image data of the target mechanical equipment in the mechanical equipment operation area;

[0128] S3. Perform parallel inference on the image data of the operator's work area;

[0129] S4. Determine whether the target machinery or personnel have entered the risk area, so as to generate a corresponding early warning or alarm signal.

[0130] Figure 8 The flowchart of an obstacle avoidance method based on computer vision according to an embodiment of the present invention is disclosed, combined with... Figure 7 and Figure 8 This section details obstacle avoidance methods based on computer vision.

[0131] S1. Acquire port image data, which includes image data of machinery and equipment and the work area of ​​workers.

[0132] The image acquisition module obtains real-time video streams of the work area from a top-down camera fixed to the gantry crane's trunk. This image data simultaneously includes the dynamically operating machinery and personnel. After input, the system first performs preprocessing and quality diagnosis on the image to eliminate invalid images caused by obstructions, overexposure, rain, fog, etc.

[0133] S2. Infer the image data of the target mechanical equipment in the mechanical equipment operation area.

[0134] In this step, the first visual detection model (OBB model) is loaded through the target machinery detection module to infer the image data of the target machinery in the machinery operation area. This model infers each input frame and is specifically used to detect grab buckets and conveyors (such as excavators), and critically outputs their rotated bounding boxes (OBBs) and corresponding attitude angles, solving the problem that horizontal bounding boxes cannot represent attitude.

[0135] S3. Perform parallel inference on the image data of the operator's work area.

[0136] At the same time, the worker detection module loads the second visual detection model (HBB model), performs parallel inference on the same frame, specifically detects workers, and outputs their horizontal bounding boxes (HBB).

[0137] The data output from the two detection modules is fed into the target stabilization module for stabilization. Through multi-frame matching, counting filtering, and a specialized smoothing algorithm for the grab angle, jitter and false alarms from single-frame detection are eliminated, outputting stable and continuous target trajectory information, providing reliable input for subsequent risk assessment.

[0138] S4. Determine whether the target machinery or personnel have entered the risk area, so as to generate a corresponding early warning or alarm signal.

[0139] In this step, the stabilized grab bucket OBB information is received. Based on the grab bucket's real-time position, size, and key angle information, concentric risk zones (e.g., inner alarm zone, outer warning zone) are dynamically generated. Then, it is determined whether a stable flow machine or personnel target has intruded into these zones. If an intrusion occurs, a warning or alarm signal is immediately generated, and key information such as the grab bucket angle is prepared to be transmitted.

[0140] The signals generated by the risk assessment module are written to the designated registers of the industrial PLC in real time and automatically via the Modbus TCP protocol. After receiving the signals, the PLC can immediately drive equipment such as gantry cranes and conveyors to execute predetermined safety strategies, such as audible and visual alarms, automatic deceleration, or emergency shutdown, thus forming a highly efficient automated safety closed loop.

[0141] The obstacle avoidance system and method based on computer vision provided by this invention have the following beneficial effects:

[0142] 1) By adopting rotating target detection technology, the real-time attitude angle of mechanical equipment such as grabs can be accurately obtained, laying the foundation for subsequent risk assessment based on the real contour.

[0143] 2) Based on the detected rotating rectangle of the grab bucket, a risk area concentric with its posture is dynamically generated, making the delineation of the danger zone more in line with the actual outline of the equipment, which greatly improves the accuracy of the early warning.

[0144] 3) Through multi-frame stabilization algorithm, jitter, false detection and noise in single-frame detection are effectively filtered out. In particular, key angle information is specially smoothed to ensure the stability and reliability of system output.

[0145] 4) A dual-model parallel inference strategy was adopted, which optimized the model scale and annotation method for large mechanical targets and small personnel targets respectively. While ensuring the accuracy of mechanical targets, it also improved the detection performance of small personnel targets.

[0146] 5) Seamless integration with industrial PLC systems via Modbus TCP protocol enables a closed loop from visual perception to safety control, and a fault detection and adaptive mechanism is designed to ensure long-term stable operation of the system in complex industrial environments.

[0147] It should be understood that the embodiments described above are merely illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or combinations thereof.

[0148] Some aspects of this application can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "system." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this application may manifest as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compressed CDs, digital multifunction DVDs, etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0149] A computer-readable medium may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and so on, or suitable combinations thereof. A computer-readable medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer-readable medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signals, or similar media, or any combination of the above media.

[0150] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0151] Furthermore, the terms "upper," "lower," "left," "right," "top," "bottom," "horizontal," and "vertical" used in the following description should be understood as the orientations shown in the relevant paragraphs and accompanying drawings. These relative terms are for illustrative purposes only and do not imply that the described apparatus must be manufactured or operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0152] It is understood that although terms such as "first," "second," and "third" may be used herein to describe various components, regions, layers, and / or parts, these components, regions, layers, and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers, and / or parts. Therefore, the first components, regions, layers, and / or parts discussed below may be referred to as second components, regions, layers, and / or parts without departing from some embodiments of the present invention.

[0153] The foregoing description of this disclosure is intended to enable any person skilled in the art to make or use it. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A computer vision-based obstacle avoidance system, characterized in that, The system includes at least an image acquisition module, a target machinery and equipment detection module, a worker detection module, and a risk assessment module. The image acquisition module is used to acquire port image data, which includes image data of mechanical equipment and the work area of ​​operators; The target mechanical equipment detection module integrates a first visual detection model, and performs inference detection on the target mechanical equipment image data in the mechanical equipment operation area based on the first visual detection model to obtain a first detection result. The worker detection module integrates a second visual detection model, and performs parallel inference detection on the image data of the worker's work area based on the second visual detection model to obtain a second detection result. as well as The risk assessment module is communicatively connected to the target machinery and equipment detection module and the operator detection module. Based on the first detection result and the second detection result, it determines whether the target machinery and equipment or the operator has entered the risk area, so as to generate a corresponding early warning or alarm signal.

2. The obstacle avoidance system based on computer vision according to claim 1, characterized in that, The first visual detection model uses a rotated rectangle to detect the image data of the target mechanical equipment and outputs a first detection result; The second visual detection model uses a horizontal rectangular bounding box to detect the image data of the worker's work area and outputs a second detection result.

3. The obstacle avoidance system based on computer vision according to claim 2, characterized in that, The first visual detection model was trained on a dataset of target mechanical equipment that includes rotated bounding box annotations; The second visual detection model was trained on a people dataset containing horizontal rectangular labels.

4. The obstacle avoidance system based on computer vision according to claim 2, characterized in that, The risk assessment module generates at least one concentric enlarged area as a risk area based on the rotating rectangular frame of the mechanical equipment. The concentric magnification region includes a first magnification region and a second magnification region that are concentric with the rotating rectangular frame of the mechanical equipment, wherein the magnification factor of the first magnification region is less than that of the second magnification region.

5. The obstacle avoidance system based on computer vision according to claim 2, characterized in that, It also includes a target stabilization module: The target stabilization module is communicatively connected to the mechanical equipment target detection module and the operator detection module, and uses a multi-frame stabilization algorithm to perform time-series stabilization processing on the first detection result and the second detection result.

6. The obstacle avoidance system based on computer vision according to claim 5, characterized in that, The target stabilization module performs time-series stabilization processing on the first detection result and the second detection result, further including: Based on the set minimum number of active frames, a smoothing algorithm is used to filter the first and second detection results; By calculating the state changes of the target between consecutive frames, mutation detection results exceeding a preset threshold are filtered out; Output the stabilized target information.

7. The obstacle avoidance system based on computer vision according to claim 1, characterized in that, The image acquisition module further includes: analyzing the quality of the image data to detect whether the image acquisition module has malfunctioned; In response to the detection of a malfunction in the image acquisition module, the image processing parameters are adjusted or the system switches to a backup image acquisition module.

8. The obstacle avoidance system based on computer vision according to claim 1, characterized in that, It also includes an industrial communication module: The industrial communication module is communicatively connected to the risk assessment module and is used to communicate with the programmable logic controller of the port equipment via the TCP protocol to transmit the early warning or alarm signal to the programmable logic controller.

9. The obstacle avoidance system based on computer vision according to claim 2, characterized in that, The target mechanical equipment is a mobile mechanical equipment.

10. A computer vision-based obstacle avoidance method, said method operating on the system described in any one of claims 1 to 9, characterized in that, The method includes: Acquire port image data, including image data of machinery and equipment and the work areas of workers; Inference is performed on the image data of the target mechanical equipment in the mechanical equipment operation area; Parallel inference is performed on the image data of the worker's work area; Determine whether the target machinery or personnel have intruded into the risk area, and generate a corresponding warning or alarm signal.

Images