Obstacle warning method and system for blind area in front of crane, crane and medium

Abstract

This invention relates to the field of crane technology, and discloses a method, system, crane, and medium for obstacle warning in the blind spot in front of a crane. Applied to cranes, at least one camera is installed in front of the crane's cab. The method includes: acquiring detection images obtained by each camera detecting the area in front of the crane; detecting image features of each detection image to obtain obstacle features of target obstacles and the corresponding obstacle type in each detection image; extracting target feature points of the target obstacle from the obstacle features, and determining the target distance between the target obstacle and the crane based on the target feature points; determining the target risk level according to the type of target obstacle and the target distance, and controlling the crane to perform corresponding warning processing based on the target risk level. This method enables accurate detection of obstacles in the blind spot in front of the crane caused by the crane hook obstructing the view, and effective warning of collision risks, ensuring the safe operation of the crane.

Description

Technical Field

[0001] This invention relates to the field of crane technology, specifically to a method, system, crane, and medium for early warning of obstacles in the blind spot in front of a crane. Background Technology

[0002] Currently, cranes, as special-operation equipment, often have various types of hooks suspended in front of them during operation. However, these hooks can obstruct the driver's view, creating blind spots. While existing blind spot detection methods used in passenger cars can address this issue, they are prone to misidentifying hooks as obstacles or failing to recognize obstacles obscured by the hooks, thus failing to solve the problem of identifying obstructions in crane hook-obstructed blind spots. Furthermore, the diverse forms of existing crane hooks exacerbate the difficulty of obstacle identification in blind spots, leading to inaccurate identification of obstacles such as pedestrians or other vehicles, which can easily result in collisions and pose a significant risk of accidents to cranes. Therefore, there is an urgent need for a universal and accurate obstacle warning method specifically designed for cranes with obstructed hooks, to overcome the problems of blind spot identification and collision detection when crane hooks obstruct obstacles. Summary of the Invention

[0003] In view of this, the present invention provides a method, system, crane and medium for obstacle warning in the blind spot in front of a crane, to solve the problem that the prior art ignores obstacle warning in the blind spot of the crane, making it difficult to accurately detect obstacles in the blind spot and effectively warn of collision risks, thus failing to ensure the safe operation of the crane.

[0004] In a first aspect, the present invention provides an obstacle warning method for the blind spot in front of a crane, applied to a crane, wherein at least one camera is installed in front of the crane's cab, and the method includes:

[0005] Acquire detection images obtained by each camera detecting the front of the crane;

[0006] The image features of each detected image are detected to obtain the obstacle features of the target obstacle and the obstacle type corresponding to the target obstacle in each detected image;

[0007] Extract target feature points of the target obstacle from the obstacle features, and determine the target distance between the target obstacle and the crane based on the target feature points;

[0008] The target risk level is determined based on the type of obstacle and the distance to the target, and the crane is controlled to carry out corresponding early warning measures based on the target risk level.

[0009] This invention uses at least one camera installed in front of the crane cab to acquire a detection image in front of the crane, performs obstacle detection on the detection image, calculates the target distance between the target obstacle and the crane based on the obstacle characteristics and corresponding obstacle type, and determines the target risk level based on the type of target obstacle and the target distance, thereby controlling the crane to perform corresponding early warning processing. It can provide a universal and effective obstacle early warning method in the crane's blind spot, overcome the problem of obstacle blind spot recognition and collision detection caused by the crane hook obstruction, and help ensure the safe operation of the crane.

[0010] In one optional implementation, extracting target feature points of the target obstacle from obstacle features and determining the target distance between the target obstacle and the crane based on the target feature points includes:

[0011] Similarity calculation is performed on the detected images from each camera;

[0012] When the similarity results meet the preset similarity conditions, the obstacle features in each detection image are matched to obtain the target feature points of the target obstacle in each detection image.

[0013] Calculate the relative distance between each target feature point and the crane, and determine the target distance between the target obstacle and the crane based on multiple relative distances.

[0014] This invention calculates the similarity of camera detection images obtained from different perspectives. When the similarity results meet preset similarity conditions, obstacle feature matching is performed on each detection image. Then, based on the matching results, the target feature points of the target obstacle in each detection image are determined. Based on the relative distance between each target feature point and the crane, the target distance between the target obstacle and the crane is determined. This ensures the accuracy of the target distance calculation and improves the obstacle recognition accuracy.

[0015] In one optional implementation, the target risk level is determined based on the type of the target obstacle and the target distance, and the crane is controlled to perform corresponding early warning processing based on the target risk level, including:

[0016] Based on the type of the target obstacle and the preset risk area, the risk areas of the target obstacle at various levels are determined. The preset risk area marks multiple risk areas corresponding to different types of obstacles.

[0017] The target level risk area is determined from the target distance of the target obstacle from the various risk areas of the target obstacle;

[0018] The target risk level of the target obstacle is determined based on the target risk area, and the crane is controlled to activate audible and visual alarms or light alarms based on the target risk level.

[0019] This invention adaptively defines different risk zones corresponding to the types of obstacles. Based on the actual detected obstacle types and their distance from the crane, it determines the target risk zone and target risk level of the current obstacle, and performs corresponding audible and visual alarms or light alarms based on the target risk level. This enables graded processing of collision risks of different vehicles and improves the early warning efficiency of obstacles in the blind spot in front of the crane.

[0020] In one optional implementation, before controlling the crane to perform corresponding early warning processing based on the target risk level, the obstacle early warning method for the blind spot in front of the crane further includes:

[0021] Calculate multiple target distances corresponding to the target obstacle within a consecutive preset number of frames, and obtain the relative speed between the target obstacle and the crane for each frame.

[0022] The relative position change trend of the target obstacle is determined based on multiple target distances, and the relative velocity change trend of the target obstacle is determined based on multiple relative velocities;

[0023] The target risk level of the obstacle is adjusted based on the trends of relative position and relative velocity changes.

[0024] This invention takes into account the occasional errors in obstacle risk level determination. It calculates multiple target distances corresponding to the same obstacle and obtains the relative speed between the target obstacle and the crane corresponding to the corresponding frame number by using the detection images obtained by continuous multi-frame detection from the camera. Based on the multiple target distances and relative speeds, it determines the relative position change trend and relative speed change trend of the obstacle, and adjusts the target risk level of the obstacle based on the relative position change trend and relative speed change trend. This can improve the accuracy of obstacle risk level determination and further ensure the accuracy and effectiveness of obstacle risk warning.

[0025] In one optional implementation, adjusting the target risk level of the target obstacle based on the relative position change trend and the relative velocity change trend includes:

[0026] If the relative position change trend is greater than 0 and the relative velocity change trend is greater than 0, the target risk level of the target obstacle will be reduced until the preset minimum risk level is reached and the level adjustment will stop.

[0027] If the relative position change trend is greater than 0 and the relative velocity change trend is less than 0, or if the relative position change trend is less than 0 and the relative velocity change trend is greater than 0, then the target risk level of the target obstacle remains unchanged.

[0028] If the relative position change trend is less than 0 and the relative velocity change trend is greater than 0, the target risk level of the obstacle will be increased until the preset highest risk level is reached and the level adjustment will stop.

[0029] This invention determines the corresponding risk level adjustment strategy by analyzing the relationship between the relative position change trend and the relative velocity change trend of obstacles and 0, which can greatly ensure the accuracy and rationality of obstacle risk level determination.

[0030] In one optional implementation, a trained detection model is used to detect image features of each detection image, wherein the training process of the detection model includes:

[0031] Obtain a training dataset that contains at least hook occlusion data for different obstacles;

[0032] The pre-set detection model is trained based on the training dataset to obtain a trained detection model.

[0033] This invention trains a detection model using data containing hook occlusions of different obstacles, ensuring accurate identification of image features in each detected image and further improving the accuracy and efficiency of obstacle warning in the blind spot in front of the crane caused by hook occlusion.

[0034] In one alternative implementation, obtaining the training dataset includes:

[0035] We collected the first dataset of each camera in scenarios where the crane is suspended with different types of hooks, and the second dataset of each camera in scenarios where the crane is not suspended with hooks.

[0036] Extract the hook image for each data point in the first dataset to obtain multiple hook images;

[0037] For each data point in the second dataset, obstacle identification is performed to obtain multiple identification results, where each identification result contains at least one obstacle.

[0038] By randomly covering obstacles in any data in the second dataset with each hook image, multiple hook occlusion data are obtained. All hook occlusion data are then added to the second dataset to obtain the training dataset.

[0039] This invention takes into account the different types of crane hooks in practical applications. By randomly mixing images of different crane hook models with images of unobstructed obstacles to generate hook occlusion data, and then constructing a training dataset based on the generated hook occlusion data, the diversity of samples in the training dataset can be guaranteed, further improving the accuracy of the detection model in identifying various obstacles.

[0040] Secondly, the present invention provides an obstacle warning system for the blind spot in front of a crane, applied to a crane, wherein at least one camera is installed in front of the crane's cab, and the system includes:

[0041] The data acquisition module is used to acquire detection images obtained by each camera from detecting the front of the crane;

[0042] The target detection module is used to detect the image features of each detection image, and obtain the obstacle features of the target obstacle and the obstacle type corresponding to the target obstacle in each detection image;

[0043] The location determination module is used to extract target feature points of the target obstacle from obstacle features, and determine the target distance between the target obstacle and the crane based on the target feature points;

[0044] The risk warning module is used to determine the target risk level based on the type of target obstacle and the target distance, and to control the crane to perform corresponding warning processing based on the target risk level.

[0045] The obstacle warning system for blind spots in front of a crane of the present invention is applied to cranes with at least one camera installed in front of the cab. By detecting the detection images obtained by each camera in front of the crane, the system obtains detection results containing the features of different obstacles and their corresponding types. Based on the detection results, the distance between the obstacle and the crane is calculated, and the risk level is determined based on the type and distance of the obstacle, thereby controlling the crane to perform corresponding warning processing. This system can overcome the problem of obstacle blind spot identification and collision detection caused by the crane hook obstruction, and provides a universal and accurate obstacle warning method for situations where the crane hook is obstructed in front of the crane, greatly improving the operational safety of the crane.

[0046] Thirdly, the present invention provides a crane, the crane including a controller, the controller including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform an obstacle warning method for the blind spot in front of the crane as described in the first aspect or any corresponding embodiment.

[0047] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute an obstacle warning method for a blind spot in front of a crane as described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0048] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating a method for early warning of obstacles in the blind spot in front of a crane according to an embodiment of the present invention;

[0050] Figure 2 This is a flowchart illustrating another obstacle warning method for the blind spot in front of a crane according to an embodiment of the present invention;

[0051] Figure 3 This is a diagram showing the camera installation;

[0052] Figure 4 This is a schematic diagram of the obstacle risk area;

[0053] Figure 5 This is a schematic diagram of a crane blind spot monitoring device;

[0054] Figure 6 This is a structural block diagram of an obstacle warning system for the blind spot in front of a crane according to an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the controller of the crane according to an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This invention provides an embodiment of an obstacle warning method for blind spots in front of a crane. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0058] This embodiment provides an obstacle warning method for the blind spot in front of a crane, applied to a crane, wherein at least one camera is installed in front of the crane's cab. Figure 1This is a flowchart illustrating an obstacle warning method for the blind spot in front of a crane according to an embodiment of the present invention, as shown below. Figure 1 As shown, the process includes the following steps:

[0059] Step S101: Obtain the detection images obtained by each camera detecting the front of the crane.

[0060] It should be noted that in this embodiment, the camera is used to acquire environmental images in front of the crane. The specific model and number of cameras are not limited here, and can be determined according to actual needs.

[0061] Step S102: Detect the image features of each detection image to obtain the obstacle features of the target obstacle and the obstacle type corresponding to the target obstacle in each detection image.

[0062] In this embodiment, the specific detection method for image features and the type of obstacle are not limited and can be adaptively adjusted according to actual needs. For example, the Scale-Invariant Feature Transform (SIFT) algorithm can be used (its main feature is that the algorithm is invariant to changes in image scale, rotation, and brightness, and can extract feature points with uniqueness and stability, thereby achieving efficient image matching and recognition); obstacle types include pedestrians, other vehicles, etc., which are only used as examples.

[0063] Step S103: Extract target feature points of the target obstacle from the obstacle features, and determine the target distance between the target obstacle and the crane based on the target feature points.

[0064] In this embodiment, target feature points represent the corresponding characteristics of target obstacles, and their specific content is adaptively adjusted according to the obstacle type and feature detection method.

[0065] Step S104: Determine the target risk level based on the type of target obstacle and the target distance, and control the crane to perform corresponding early warning processing based on the target risk level.

[0066] It should be noted that this embodiment considers the different types of obstacles and their relative distance from the crane to determine the risk level, and performs corresponding early warning processing on the crane according to different risk levels. The specific classification of risk levels and the specific methods of early warning processing can be adaptively adjusted according to actual needs. For example, risk levels include high-risk and low-risk levels. If the obstacle is a pedestrian, the collision risk to the crane is low, and the corresponding target distance value is larger for a high-risk level. When the obstacle is another vehicle, since vehicles pose a higher collision risk to the crane than pedestrians, the corresponding target distance value for other vehicles is smaller for a high-risk level. For high-risk levels, the crane can be controlled to perform audible and visual alarm early warning processing, which is only an example.

[0067] The obstacle warning method for blind spots in front of a crane according to embodiments of the present invention acquires a detection image of the area in front of the crane using at least one camera installed in front of the crane cab. Obstacles are detected in the detection image. The target distance between the target obstacle and the crane is calculated based on the obstacle characteristics and corresponding obstacle type. The target risk level is determined based on the type of the target obstacle and the target distance, and the crane is then controlled to perform corresponding warning processing. This method provides a universal and effective warning method for obstacles in the crane's blind spot, which can overcome the problem of obstacle blind spot identification and collision detection caused by the crane hook obstruction, and helps to ensure the safe operation of the crane.

[0068] This embodiment provides an obstacle warning method for the blind spot in front of a crane, applied to a crane, wherein at least one camera is installed in front of the crane's cab. Figure 2 This is a flowchart illustrating another obstacle warning method for the blind spot in front of a crane according to an embodiment of the present invention, as shown below. Figure 2 As shown, the process includes the following steps:

[0069] Step S201: Acquire detection images obtained from the cameras detecting the area in front of the crane. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0070] Step S202: Detect the image features of each detection image to obtain the obstacle features of the target obstacle and the obstacle type corresponding to the target obstacle in each detection image.

[0071] In this embodiment, a trained detection model is used to detect image features of each detection image. The training process of the detection model includes:

[0072] Step A1: Obtain the training dataset, which contains at least hook occlusion data for different obstacles.

[0073] Specifically, obtaining the training dataset in step A1 above includes:

[0074] Step A11: Collect the first dataset of each camera corresponding to the crane in scenarios where different types of hooks are suspended, and the second dataset of each camera corresponding to the crane in scenarios where no hook is suspended.

[0075] Step A12: Extract hook images from each data point in the first dataset to obtain multiple hook images.

[0076] In this embodiment, the specific method for extracting the hook image is not limited, and it can be adaptively obtained according to conventional image segmentation algorithms in the field, such as the U-Net semantic segmentation algorithm based on convolutional neural networks.

[0077] Step A13: Perform obstacle identification on each data point in the second dataset to obtain multiple identification results, where each identification result contains at least one obstacle.

[0078] In this embodiment, the specific method of obstacle identification is not limited, and can be adaptively obtained according to conventional image detection algorithms in the field, such as the SSD (Single Shot MultiBox Detector) target detection algorithm.

[0079] Step A14: Use the images of each hook to randomly cover the obstacles of any data in the second dataset to obtain multiple hook occlusion data, and add all the hook occlusion data to the second dataset to obtain the training dataset.

[0080] In this embodiment, the hook image is used as a foreground occluder to cover various obstacles according to random positions and proportions, thereby obtaining hook occlusion data for different obstacles.

[0081] In this embodiment of the invention, considering the different types of crane hooks in practical applications, images of different crane hook models are randomly mixed with images of unobstructed obstacles to generate hook occlusion data. Then, a training dataset is constructed based on the generated hook occlusion data, which can ensure the diversity of samples in the training dataset and further improve the accuracy of the detection model in identifying various obstacles.

[0082] Step A2: Train the preset detection model based on the training dataset to obtain the trained detection model.

[0083] It should be noted that the specific type of the preset detection model in this embodiment is not limited and can be adaptively adjusted according to actual needs. Specifically, this embodiment uses hook occlusion data containing different obstacles to train the model and obtain a trained detection model, which can ensure the accurate identification of image features in each detection image, and further improve the accuracy and efficiency of obstacle warning in the blind spot in front of the crane caused by hook occlusion.

[0084] Step S203: Extract target feature points of the target obstacle from the obstacle features, and determine the target distance between the target obstacle and the crane based on the target feature points.

[0085] Specifically, step S203 includes:

[0086] Step S2031: Calculate the similarity of the detected images from each camera.

[0087] In this embodiment, the specific calculation method for detecting image similarity is not limited. For example, commonly used similarity indicators, such as Euclidean distance and cosine similarity, can be used to analyze the similarity of images.

[0088] Step S2032: When the similarity result meets the preset similarity conditions, the obstacle features in each detection image are matched to obtain the target feature points of the target obstacle in each detection image.

[0089] In this embodiment, the specific content of the preset similarity conditions is adaptively determined according to the similarity calculation method adopted. For example, if the similarity of images is calculated using Euclidean distance, since the smaller the Euclidean distance, the greater the similarity, the corresponding preset similarity condition is the set distance threshold, which is only used as an example.

[0090] Step S2033: Calculate the relative distance between each target feature point and the crane, and determine the target distance between the target obstacle and the crane based on multiple relative distances.

[0091] In this embodiment, the method of determining the target distance by multiple relative distances is not limited, but is adapted to actual needs. For example, the average value of multiple relative distances can be calculated and used as the target distance. This is only an example.

[0092] In this embodiment of the invention, similarity calculation is performed on camera detection images obtained from different perspectives. When the similarity result meets the preset similarity conditions, obstacle feature matching is performed on each detection image. Then, the target feature points of the target obstacle in each detection image are determined based on the matching result. The target distance between the target obstacle and the crane is determined based on the relative distance between each target feature point and the crane. This ensures the accuracy of the target distance calculation and improves the obstacle recognition accuracy.

[0093] Step S204: Determine the target risk level based on the type of target obstacle and the target distance, and control the crane to perform corresponding early warning processing based on the target risk level.

[0094] Specifically, step S204 includes:

[0095] Step S2041: Determine the risk areas of the target obstacle at various levels based on the type of the target obstacle and the preset risk areas. The preset risk areas mark multiple risk areas corresponding to different types of obstacles.

[0096] Step S2042: Determine the target level risk area from the various risk areas of the target obstacle based on the target distance of the target obstacle.

[0097] Step S2043: Determine the target risk level of the target obstacle based on the target risk area, and control the crane to perform an audible and visual alarm or a light alarm based on the target risk level.

[0098] In this embodiment of the invention, different risk levels are adaptively defined for each obstacle type. Based on the actual detected obstacle type and its distance from the crane, the target risk level of the current obstacle is determined and its target risk level is determined. Based on the target risk level, corresponding audible and visual alarms or light alarms are triggered. This enables graded processing of collision risks for different vehicles and improves the early warning efficiency of obstacles in the blind spot in front of the crane.

[0099] In practical applications, due to the occasional errors in determining the obstacle risk level—that is, in this embodiment, the target risk level corresponding to the target obstacle is determined based on single-frame image data from at least one camera, and there are errors in the recognition of single-frame image data—this embodiment also includes a risk level adjustment process to improve the accuracy of crane obstacle warning. Therefore, before controlling the crane to perform corresponding warning processing based on the target risk level, the obstacle warning method for the blind spot in front of the crane in this embodiment further includes:

[0100] Step B1: Calculate multiple target distances corresponding to the target obstacle within a consecutive preset number of frames, and obtain the relative speed between the target obstacle and the crane for each frame.

[0101] In this embodiment, the specific value of the preset frame number is adaptively adjusted according to actual needs, such as a preset frame number of 5 frames.

[0102] Step B2: Determine the relative position change trend of the target obstacle based on multiple target distances, and determine the relative velocity change trend of the target obstacle based on multiple relative velocities.

[0103] Step B3: Adjust the target risk level of the target obstacle based on the trends of relative position change and relative velocity change.

[0104] Specifically, step B3 above includes:

[0105] Step B31: If the relative position change trend is greater than 0 and the relative velocity change trend is greater than 0, then reduce the target risk level of the target obstacle until the preset minimum risk level is reached and the level adjustment stops.

[0106] Step B32: If the relative position change trend is greater than 0 and the relative velocity change trend is less than 0, or the relative position change trend is less than 0 and the relative velocity change trend is greater than 0, then the target risk level of the target obstacle remains unchanged.

[0107] Step B33: If the relative position change trend is less than 0 and the relative velocity change trend is greater than 0, then increase the target risk level of the target obstacle until the preset highest risk level is reached and the level adjustment stops.

[0108] It should be noted that the specific content of the preset minimum and maximum risk levels in this embodiment can be adaptively set according to actual needs. Specifically, by analyzing the relationship between the relative position and relative velocity change trends of obstacles and 0, the corresponding risk level adjustment strategy can be determined, which can greatly ensure the accuracy and rationality of obstacle risk level determination.

[0109] In this embodiment of the invention, considering the occasional errors in obstacle risk level determination, the detection images obtained by continuous multi-frame detection by the camera are used to calculate multiple target distances corresponding to the same target obstacle and to obtain the relative speed between the target obstacle and the crane corresponding to the corresponding frame number. Based on the multiple target distances and relative speeds, the relative position change trend and relative speed change trend of the obstacle are determined respectively. Based on the relative position change trend and relative speed change trend, the target risk level of the obstacle is adjusted, which can improve the accuracy of obstacle risk level determination and further ensure the accuracy and effectiveness of obstacle risk warning.

[0110] In one specific embodiment, a crane blind spot monitoring and collision warning scheme is provided to realize blind spot monitoring and collision object identification during crane movement with hook, provide early warning of collision risks, and improve crane operation safety. Specifically, the specific process of the above scheme includes:

[0111] 1. Install a camera.

[0112] In this embodiment, refer to Figure 3Two cameras (i.e., ordinary cameras) are installed on the left and right sides in front of the crane's cab. The two cameras fill in the blind spots for each other. The camera installation positions only need to cover the blind spots blocked by the hook as much as possible. In other words, the camera installation positions can reduce the obstruction of the view by the front hook.

[0113] 2. Delineate risk areas.

[0114] In this embodiment, different risk zones are defined for obstacles including four types of dynamic obstacles: pedestrians, cyclists, small vehicles, and large vehicles. Specifically, refer to [link to relevant documentation]. Figure 4 The first-level risk area is 0-d(a / b / c / d)1, the second-level risk area is d(a / b / c / d)1-d(a / b / c / d)2, and the third-level risk area is d(a / b / c / d)2-d(a / b / c / d)3; where a / b / c / d represent four types of dynamic obstacles respectively. For example, the first-level risk area for a cyclist is 0-db1, the second-level risk area is db1-db2, and the third-level risk area is db3-db4. This is only an example for illustration.

[0115] 3. Construct a blind spot monitoring device in front of the crane.

[0116] In this embodiment, Figure 5 This is a schematic diagram of a crane blind spot monitoring device. Figure 5 It can be seen that the device consists of two phases: I. Training phase and II. Application phase. The device is specifically composed of a target ranging module, a target detection module, a data generation module, and a decision module. The workflow of each module in different phases is shown below.

[0117] During the training phase, to quickly generate a dataset of hook occlusions with various obstacles, the data generation module employs a method of randomly mixing and synthesizing images of unoccluded obstacles with images of hooks. Specifically, first, dataset A is collected from the perspectives of two cameras in front of the cab for different types of hooks, and dataset B is collected from the perspectives of two cameras in front of the cab for different obstacles in hookless scenes. The hooks in dataset A are cropped out as foreground occluders and used to cover the obstacles in dataset B at random positions and proportions, generating a synthetic dataset C. The synthetic dataset C is then used to train the object detection model in the object detection module.

[0118] In the application phase, the device is integrated into the crane, and three modules—target detection, target ranging, and decision-making—run within the onboard controller (i.e., the crane's main controller). Specifically, firstly, real-time data from two front-facing cameras is input into the target detection module. Two target detection models (such as YOLOv5) are used to detect targets from the two camera feeds, and the detected obstacle type is output to the decision-making module. This involves outputting the image within the bounding box of the detected obstacle to the target ranging module. Next, the target ranging module extracts and matches features from the bounding boxes output from the two different cameras. Based on binocular stereo vision measurement technology, the module calculates the relative position of the target obstacle to the crane using successfully matched feature points and outputs this obstacle position information to the decision-making module. Finally, the decision-making module outputs different levels of warning information based on the obstacle type and location. For example, for vulnerable road users such as pedestrians and cyclists at close range, a high-level alarm signal (e.g., audible and visual alarm) is given; for vulnerable road users or vehicles at greater distances, a low-level alarm signal (e.g., light alarm) is given.

[0119] In this embodiment, a specific implementation of a crane blind spot monitoring and collision warning scheme is provided, including:

[0120] 1. Obtain the occlusion-mixed dataset and train the model. The specific process includes:

[0121] Step 1: Collect image data of different types of hooks hanging in front of the crane while it is in motion, and use image segmentation algorithms (such as U-Net and SegNet) to extract the hook images.

[0122] Step 2: Collect image data of the crane in motion with no hook hanging in front and different types of obstacles (such as pedestrians, bicycles, vehicles, etc.), and use a target detection algorithm (such as YOLOv5) to obtain the obstacle bounding boxes.

[0123] Step 3: Based on the position and size of the envelope, the hook image obtained in Step 1 is used as the foreground and mixed with the obstacle data collected in Step 2. This allows the hook image to occlude obstacles in the vicinity of the envelope at random positions and proportions, generating a synthetic dataset with different obstacle occlusion situations.

[0124] Step 4: Mix the image dataset collected in Step 2 with the synthetic dataset obtained in Step 3 to train an object detection model (such as YOLOv).

[0125] 2. The process of detecting obstacle targets, extracting target image feature points, and calculating the target position based on the feature points from the two cameras includes:

[0126] Step 1: Port the trained object detection model to the crane controller, and perform object detection using one object detection model for each of the image data collected by the two cameras. It should be noted that this embodiment can also use a single object detection model, which needs to be trained using image data from both cameras.

[0127] Step 2: Output the types of obstacles detected in the two image data to the decision module, and output the images within the envelope of the detected obstacles to the target ranging module.

[0128] Step 3: Extract features from the bounding boxes of the target detection results from the two cameras (e.g., using SIFT algorithm, Harris algorithm, etc.).

[0129] Step 4: Compare the similarity of the feature descriptors of the two image data obtained in Step 1 (e.g., use Euclidean distance for similarity measurement). When the similarity is greater than a set threshold, the pixels in the two image data can be matched to obtain a set of matched pixels.

[0130] Step 5: Based on binocular stereo vision measurement technology, calculate the position of the matching feature points relative to the crane based on the matching pixel point set to form an obstacle feature point cloud, and take the centroid position of the obstacle feature point cloud as the obstacle position.

[0131] 3. Alarm decision-making, the specific process includes:

[0132] Step 1: Establish a risk assessment model by using the relative position of the obstacle and the crane, and the type of obstacle, and score the obstacle collision risk.

[0133] Step 2: Calculate the relative positional change trend of the obstacle and the crane in the most recent consecutive frames (e.g., 8 frames). If they are relatively close, increase the risk level score; if they are relatively far apart, decrease the risk level score.

[0134] Step 3: Output different forms of alarms based on the risk level score, such as sound and light alarms for high-risk levels and light alarms for low-risk levels.

[0135] It should be noted that, for reference Figure 4 This embodiment includes four obstacle types: pedestrians, cyclists, small vehicles, and large vehicles. Based on the risk zone where the obstacle is first detected, different initial risk levels are assigned. Subsequently, the relative distance change Δd between the same obstacle and the crane in the direction of travel and the average speed v of the obstacle relative to the vehicle are detected within several consecutive frames. The risk level is determined based on the relative distance change Δd and the average speed v (i.e., the risk level is jointly determined by the alarm range, the relative distance change Δd between the obstacle and the vehicle, and the average speed v of the obstacle relative to the vehicle). Specifically:

[0136] 1) If Δd>0 and v>0 (i.e. the obstacle is relatively far away and will continue to move away), reduce the risk level based on the current risk level of the obstacle's location until the lowest risk level is reached.

[0137] 2) If Δd>0 and v<0 (i.e., the obstacle is relatively far away but will move closer later) or Δd<0 and v>0 (i.e., the obstacle is relatively close but will move away later), maintain the risk level of the area where the obstacle is currently located.

[0138] 3) If Δd<0 and v<0 (the obstacle is relatively close and will continue to approach), increase the risk level based on the current risk level of the area where the obstacle is located until the highest risk level is reached.

[0139] In one specific embodiment, see Figure 4 It is known that the cyclist's level 1 risk zone is 0-db1, level 2 risk zone is db1-db2, and level 3 risk zone is db3-db4. Specifically, when the cyclist crosses the level 2 risk zone, theoretically both Δd and v are 0, and the risk level of this zone remains unchanged. In this case, the crane will continue to issue a level 2 alarm. However, when the cyclist travels in the same direction as the vehicle in the level 2 risk zone, if Δd < 0 and v < 0, even if the cyclist's area is a level 2 risk zone, the risk level will be upgraded to level 1; if Δd > 0 and v > 0, even if the cyclist's area is a level 2 risk zone, the risk level will be downgraded to level 3.

[0140] It should be noted that the alarm procedures for different collision warning scenarios in this embodiment are as follows:

[0141] 1. Controller: Runs two target detection modules to process the video information received from the two cameras, obtaining the target's bounding box from two different perspectives. Then, it processes the data based on the warning process described in the aforementioned specific implementation. Specifically, the controller uses the aforementioned warning process to calculate the relative position of the obstacle and the vehicle. Based on the relative position of the obstacle and the vehicle, and combined with the vehicle's status information, it outputs an alarm signal according to the alarm decision scheme described in the specific implementation.

[0142] 2. Instruments: Receive alarm signals from the controller, provide light alarms for low-level warning information, and sound and light alarms for high-level alarm information.

[0143] It should be noted that when the obstacle in front of the crane is small and close to the hook, the obstacle will inevitably be obscured to some extent. The warning process described above in this embodiment, that is, the target detection method based on binocular stereo vision, can still efficiently detect the distance and type of the obstacle obscured by the hook.

[0144] In summary, the obstacle warning method for the blind spot in front of the crane according to the present invention can accurately detect obstacles in the blind spot caused by the crane hook obstructing the view, and effectively warn of collision risks, thus greatly ensuring the safe operation of the crane.

[0145] This embodiment also provides an obstacle warning system for the blind spot in front of a crane. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, a "module" can be a combination of software and / or hardware that performs a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, hardware implementations, or a combination of software and hardware, are also possible and contemplated.

[0146] This invention provides an obstacle warning system for the blind spot in front of a crane, applied to a crane, wherein at least one camera is installed in front of the crane's cab, such as... Figure 6 As shown, the system includes:

[0147] The data acquisition module 601 is used to acquire detection images obtained by each camera detecting the front of the crane.

[0148] The target detection module 602 is used to detect the image features of each detection image and obtain the obstacle features of the target obstacle and the obstacle type corresponding to the target obstacle in each detection image.

[0149] The position determination module 603 is used to extract target feature points of the target obstacle from the obstacle features and determine the target distance between the target obstacle and the crane based on the target feature points.

[0150] The risk warning module 604 is used to determine the target risk level based on the type of target obstacle and the target distance, and to control the crane to perform corresponding warning processing based on the target risk level.

[0151] In some optional implementations, the target detection module 602 includes: a first detection submodule and a second detection submodule; wherein, the first detection submodule is used to acquire a training dataset, the training dataset containing at least hook occlusion data of different obstacles; the second detection submodule is used to train a preset detection model based on the training dataset to obtain a trained detection model.

[0152] In some optional implementations, the first detection submodule includes: a first acquisition unit, a second acquisition unit, a third acquisition unit, and a fourth acquisition unit; wherein, the first acquisition unit is used to collect a first dataset of each camera corresponding to a crane in scenarios where different types of hooks are suspended, and a second dataset of each camera corresponding to a crane in scenarios where no hooks are suspended; the second acquisition unit is used to extract hook images from each data in the first dataset to obtain multiple hook images; the third acquisition unit is used to perform obstacle recognition on each data in the second dataset to obtain multiple recognition results, wherein each recognition result contains at least one obstacle; the fourth acquisition unit is used to randomly cover the obstacle of any data in the second dataset using each hook image to obtain multiple hook occlusion data, and add all hook occlusion data to the second dataset to obtain a training dataset.

[0153] In some optional implementations, the location determination module 603 includes: a first determination submodule, a second determination submodule, and a third determination submodule; wherein, the first determination submodule is used to calculate the similarity of the detected images detected by each camera; the second determination submodule is used to match the obstacle features in each detected image when the similarity result meets the preset similarity conditions, and obtain the target feature points of the target obstacle in each detected image; the third determination submodule is used to calculate the relative distance between each target feature point and the crane, and determine the target distance between the target obstacle and the crane based on multiple relative distances.

[0154] In some optional implementations, the risk warning module 604 includes: a first warning submodule, a second warning submodule, and a third warning submodule; wherein, the first warning submodule is used to determine the various levels of risk areas of the target obstacle according to the type of the target obstacle and the preset risk area, the preset risk area marking multiple levels of risk areas corresponding to different types of obstacles; the second warning submodule is used to determine the target level risk area from the various levels of risk areas of the target obstacle based on the target distance of the target obstacle; the third warning submodule is used to determine the target risk level of the target obstacle according to the target level risk area, and control the crane to perform audible and visual alarms or light alarms based on the target risk level.

[0155] In some optional implementations, the system further includes: a level adjustment module, used to calculate multiple target distances corresponding to the target obstacle within a consecutive preset number of frames, and to obtain the relative speed between the target obstacle and the crane for each frame; to determine the relative position change trend of the target obstacle based on the multiple target distances, and to determine the relative speed change trend of the target obstacle based on the multiple relative speeds; and to adjust the target risk level of the target obstacle according to the relative position change trend and the relative speed change trend.

[0156] In some optional implementations, the level adjustment module includes: an adjustment submodule, configured to: reduce the target risk level of the target obstacle if the relative position change trend is greater than 0 and the relative velocity change trend is greater than 0, until the preset minimum risk level is reached and the level adjustment stops; maintain the target risk level of the target obstacle unchanged if the relative position change trend is greater than 0 and the relative velocity change trend is less than 0, or if the relative position change trend is less than 0 and the relative velocity change trend is greater than 0; and increase the target risk level of the target obstacle if the relative position change trend is less than 0 and the relative velocity change trend is greater than 0, until the preset maximum risk level is reached and the level adjustment stops.

[0157] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0158] In this embodiment, the obstacle warning system for the blind spot in front of the crane is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0159] The obstacle warning system for the blind spot in front of the crane according to this invention can accurately detect obstacles in the blind spot caused by the crane hook obstructing the view, and effectively warn of collision risks, thus greatly ensuring the safe operation of the crane.

[0160] This invention also provides a crane, which includes a controller; please refer to [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram of the structure of the controller provided in an optional embodiment of the present invention, as shown below. Figure 7 As shown, the controller includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the main controller, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple main controllers can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 7 Take a processor 10 as an example.

[0161] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0162] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0163] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the controller. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the controller via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0164] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0165] The controller also includes a communication interface 30 for the main control chip to communicate with other devices or communication networks.

[0166] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor main control chips, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0167] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for obstacle warning in the blind spot in front of a crane, applied to a crane, wherein at least one camera is installed in front of the crane's cab, characterized in that, The method includes: Acquire detection images obtained by each camera detecting the front of the crane; The image features of each of the detected images are detected to obtain the obstacle features of the target obstacle in each of the detected images and the obstacle type corresponding to the target obstacle; Extract target feature points of the target obstacle from the obstacle features, and determine the target distance between the target obstacle and the crane based on the target feature points; The target risk level is determined based on the type of the target obstacle and the target distance, and the crane is controlled to perform corresponding early warning processing based on the target risk level; The trained detection model is used to detect image features of each of the detected images, wherein the training process of the detection model includes: Obtain a training dataset, wherein the training dataset contains at least hook occlusion data of different obstacles; The preset detection model is trained based on the training dataset to obtain a trained detection model; The acquisition of the training dataset includes: The first dataset of each camera corresponding to the crane in scenarios where different types of hooks are suspended, and the second dataset of each camera corresponding to the crane in scenarios where no hook is suspended are collected respectively. Extract hook images from each data point in the first dataset to obtain multiple hook images; For each data point in the second dataset, obstacle identification is performed to obtain multiple identification results, where each identification result contains at least one obstacle. The obstacles in any data in the second dataset are randomly covered using the images of each hook to obtain multiple hook occlusion data. All hook occlusion data are then added to the second dataset to obtain the training dataset.

2. The obstacle warning method for the blind spot in front of a crane according to claim 1, characterized in that, The step of extracting target feature points of the target obstacle from the obstacle features and determining the target distance between the target obstacle and the crane based on the target feature points includes: Similarity calculation is performed on the detected images from each of the aforementioned cameras; When the similarity result meets the preset similarity condition, the obstacle features in each of the detected images are matched to obtain the target feature points of the target obstacle in each of the detected images. Calculate the relative distance between each of the target feature points and the crane, and determine the target distance between the target obstacle and the crane based on multiple relative distances.

3. The obstacle warning method for the blind spot in front of a crane according to claim 1, characterized in that, The step of determining the target risk level based on the type of the target obstacle and the target distance, and controlling the crane to perform corresponding early warning processing based on the target risk level, includes: The target obstacle is determined according to its type and a preset risk area. The preset risk area marks multiple risk areas corresponding to different types of obstacles. Based on the target distance of the target obstacle, the target level risk area is determined from each level of risk area of ​​the target obstacle; The target risk level of the target obstacle is determined based on the target risk area, and the crane is controlled to perform an audible and visual alarm or a light alarm based on the target risk level.

4. The obstacle warning method for the blind spot in front of a crane according to claim 1, characterized in that, Before controlling the crane to perform corresponding early warning processing based on the target risk level, the method further includes: Calculate multiple target distances corresponding to the target obstacle within a consecutive preset number of frames, and obtain the relative speed between the target obstacle and the crane for each frame. The relative position change trend of the target obstacle is determined based on multiple target distances, and the relative velocity change trend of the target obstacle is determined based on multiple relative velocities; The target risk level of the target obstacle is adjusted based on the relative position change trend and the relative velocity change trend.

5. The obstacle warning method for the blind spot in front of a crane according to claim 4, characterized in that, The adjustment of the target risk level of the target obstacle based on the relative position change trend and the relative velocity change trend includes: If the relative position change trend is greater than 0 and the relative velocity change trend is greater than 0, the target risk level of the target obstacle will be reduced until the preset minimum risk level is reached and the level adjustment will stop. If the relative position change trend is greater than 0 and the relative velocity change trend is less than 0, or if the relative position change trend is less than 0 and the relative velocity change trend is greater than 0, then the target risk level of the target obstacle remains unchanged. If the relative position change trend is less than 0 and the relative velocity change trend is greater than 0, the target risk level of the obstacle will be increased until the preset highest risk level is reached and the level adjustment will stop.

6. An obstacle warning system for the blind spot in front of a crane, applied to a crane, wherein at least one camera is installed in front of the crane's cab, characterized in that, The system includes: The data acquisition module is used to acquire detection images obtained by each camera detecting the front of the crane; The target detection module is used to detect the image features of each of the detected images, and obtain the obstacle features of the target obstacle in each of the detected images and the obstacle type corresponding to the target obstacle; A location determination module is used to extract target feature points of the target obstacle from the obstacle features, and determine the target distance between the target obstacle and the crane based on the target feature points; The risk warning module is used to determine the target risk level based on the type of the target obstacle and the target distance, and control the crane to perform corresponding warning processing based on the target risk level; The trained detection model is used to detect image features of each of the detected images, wherein the training process of the detection model includes: Obtain a training dataset, wherein the training dataset contains at least hook occlusion data of different obstacles; The preset detection model is trained based on the training dataset to obtain a trained detection model; The acquisition of the training dataset includes: The first dataset of each camera corresponding to the crane in scenarios where different types of hooks are suspended, and the second dataset of each camera corresponding to the crane in scenarios where no hook is suspended are collected respectively. Extract hook images from each data point in the first dataset to obtain multiple hook images; For each data point in the second dataset, obstacle identification is performed to obtain multiple identification results, where each identification result contains at least one obstacle. The obstacles in any data in the second dataset are randomly covered using the images of each hook to obtain multiple hook occlusion data. All hook occlusion data are then added to the second dataset to obtain the training dataset.

7. A crane, characterized in that, The crane includes a controller, which includes a memory and a processor. The memory and the processor are communicatively connected to each other. The memory stores computer instructions. The processor executes the computer instructions to perform the obstacle warning method for the blind spot in front of the crane as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the obstacle warning method for the blind spot in front of the crane as described in any one of claims 1 to 5.

Images