Early warning method and system for obstacles in dead zone in front of crane, crane and medium

Abstract

The invention relates to the technical field of cranes, and discloses an obstacle early warning method and system for a dead zone in front of a crane, the crane and a medium, the method and system are applied to the crane, at least one camera is installed in front of a cab of the crane, and the method comprises the steps that detection images obtained by detecting the front of the crane through all the cameras are obtained; detecting image features of each detection image to obtain obstacle features of a target obstacle in each detection image and an obstacle type corresponding to the target obstacle; extracting a target feature point of the target obstacle from the obstacle features, and determining a target distance between the target obstacle and the crane based on the target feature point; the target risk level is determined according to the type of the target obstacle and the target distance, and the crane is controlled to perform corresponding early warning processing based on the target risk level, so that accurate detection of each obstacle in a front blind area caused by shielding of a lifting hook in front of the crane and effective early warning of the collision risk can be realized, and safe driving of the crane is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of cranes, and in particular to an obstacle warning method and system for a blind spot in front of a crane, a crane and a medium. Background Art

[0002] At present, cranes are special operation equipment, and different forms of hooks are often hung in front of them when they are driving. However, the hook will cause a certain obstruction to the driver's field of vision, thus forming a blind spot. Based on this, the existing front blind spot detection method of passenger cars is used to solve the above problems, but it is easy to misidentify the hook as an obstacle, or it is impossible to identify the obstacle blocked by the hook, and thus it is difficult to solve the problem of identifying the obstruction in the blind spot blocked by the crane hook. In addition, the various forms of existing crane hooks increase the difficulty of identifying obstacles in the blind spot in front, and thus obstacles such as pedestrians or other vehicles cannot be accurately identified, which can easily lead to collisions and bring greater risks of dangerous accidents to the crane. Therefore, there is an urgent need for a universal and accurate obstacle warning method specifically for the obstruction of the crane hook in front, so as to overcome the blind spot identification and collision detection problems of obstacles blocked by the crane hook. Summary of the invention

[0003] In view of this, the present invention provides an obstacle warning method, system, crane and medium in the blind spot in front of the crane to solve the problem that the prior art ignores the obstacle warning in the blind spot of the crane, making it difficult to accurately detect obstacles in the blind spot and effectively warn of collision risks, thereby failing to ensure the safe operation of the crane.

[0004] In a first aspect, the present invention provides an obstacle warning method for a blind spot in front of a crane, which is applied to the crane, wherein at least one camera is installed in front of a cab of the crane, and the method comprises:

[0005] Acquire detection images obtained by each camera detecting the front of the crane;

[0006] Detecting image features of each detection image to obtain obstacle features of the target obstacle in each detection image and the obstacle type corresponding to the target obstacle;

[0007] extracting a target feature point of the target obstacle from the obstacle features, and determining a target distance between the target obstacle and the crane based on the target feature point;

[0008] The target risk level is determined according to the type of target obstacle and the target distance, and the crane is controlled to perform corresponding early warning processing based on the target risk level.

[0009] The present invention obtains a detection image in front of the crane through at least one camera installed in front of the crane cab, performs obstacle detection on the detection image, calculates the target distance between the target obstacle and the crane according to the obstacle characteristics of the target obstacle and the corresponding obstacle type, and determines the target risk level based on the type of the target obstacle and the target distance, thereby controlling the crane to perform corresponding early warning processing. A universal and effective obstacle early warning method in the blind spot of the crane can be provided, which overcomes the problem of obstacle blind spot identification and collision detection caused by the occlusion of the crane hook, and helps to ensure the safe operation of the crane.

[0010] In an optional implementation, extracting a target feature point of the target obstacle from the obstacle feature, and determining a target distance between the target obstacle and the crane based on the target feature point, includes:

[0011] Calculate the similarity of the detection images detected by each camera;

[0012] When the similarity result meets the preset similarity condition, the obstacle features in each detection image are matched to obtain the corresponding target feature points of the target obstacle in each detection image;

[0013] The relative distance between each target feature point and the crane is calculated, and the target distance between the target obstacle and the crane is determined based on the multiple relative distances.

[0014] The present invention calculates the similarity of camera detection images obtained in different viewing angles, performs obstacle feature matching on each detection image when the similarity result meets a preset similarity condition, and then determines the target feature points of the target obstacle in each detection image according to the matching result, and determines the target distance between the target obstacle and the crane based on the relative distance between each target feature point and the crane, thereby ensuring the calculation accuracy of the target distance and improving the recognition accuracy of the obstacle.

[0015] In an optional implementation, a target risk level is determined according to the type of target obstacle and the target distance, and a crane is controlled to perform corresponding warning processing based on the target risk level, including:

[0016] Determine the risk areas of each level of the target obstacle according to the type of the target obstacle and the preset risk area. The preset risk area marks multiple risk areas of different levels corresponding to different types of obstacles;

[0017] Determining a target level risk area from each level risk area of ​​the target obstacle based on a target distance of the target obstacle;

[0018] The target risk level of the target obstacle is determined according to the target level risk area, and the crane is controlled to perform an audible and visual alarm or a light alarm based on the target risk level.

[0019] The present invention adaptively defines the risk areas of different levels corresponding to each obstacle by its type, determines the target level risk area and its target risk level of the current obstacle based on the obstacle type actually detected and its distance from the crane, and performs corresponding sound and light alarms or light alarms based on the target risk level, thereby realizing graded processing of collision risks of different vehicles and improving the early warning efficiency of obstacles in the blind spot in front of the crane.

[0020] In an optional implementation, before controlling the crane to perform corresponding warning processing based on the target risk level, the obstacle warning method in the blind spot in front of the crane further includes:

[0021] Calculate multiple target distances corresponding to the target obstacle within a continuous preset number of frames, and obtain the relative speed between the target obstacle and the crane corresponding to each frame number;

[0022] Determine a relative position change trend of a target obstacle based on a plurality of target distances, and determine a relative speed change trend of a target obstacle based on a plurality of relative speeds;

[0023] The target risk level of the target obstacle is adjusted according to the relative position change trend and the relative speed change trend.

[0024] The present invention takes into account occasional errors in obstacle risk level determination, calculates multiple target distances corresponding to the same target obstacle through detection images obtained through continuous multi-frame detection by a camera, and obtains the relative speeds of the target obstacle and the crane corresponding to the corresponding frame number, determines the relative position change trend and the relative speed change trend of the obstacle according to the multiple target distances and relative speeds, respectively, and adjusts the target risk level of the obstacle based on the relative position change trend and the relative speed change trend, which can improve the determination accuracy of the obstacle risk level and further ensure the accuracy and effectiveness of obstacle risk warning.

[0025] In an optional implementation, adjusting the target risk level of the target obstacle according to the relative position change trend and the relative speed change trend includes:

[0026] If the relative position change trend is greater than 0 and the relative speed change trend is greater than 0, the target risk level of the target obstacle is reduced until the level adjustment is stopped when the preset minimum risk level is reached;

[0027] If the relative position change trend is greater than 0 and the relative speed change trend is less than 0, or the relative position change trend is less than 0 and the relative speed change trend is greater than 0, the target risk level of the target obstacle remains unchanged;

[0028] If the relative position change trend is less than 0 and the relative speed change trend is greater than 0, the target risk level of the target obstacle is increased until the level adjustment is stopped when the preset highest risk level is reached.

[0029] The present invention determines the corresponding risk level adjustment strategy by analyzing the relationship between the relative position change trend and the relative speed change trend of the obstacle and 0, which can greatly ensure the accuracy and rationality of the obstacle risk level determination.

[0030] In an optional implementation, a trained detection model is used to detect image features of each detection image, wherein the training process of the detection model includes:

[0031] Obtain a training data set, where the training data set at least includes hook occlusion data of different obstacles;

[0032] The preset detection model is trained based on the training data set to obtain a trained detection model.

[0033] The present invention obtains a trained detection model by training a model using hook occlusion data containing different obstacles, which can ensure accurate recognition of image features in each detection image and further improve the accuracy and efficiency of obstacle warning in the blind spot in front of the crane due to hook occlusion.

[0034] In an optional implementation, obtaining a training data set includes:

[0035] Collecting first data sets of each camera corresponding to a scene in which the crane is hung with hooks of different models, and second data sets of each camera corresponding to a scene in which the crane is not hung with hooks;

[0036] Extracting a hook image from each data in the first data set to obtain a plurality of hook images;

[0037] Perform obstacle recognition on each data in the second data set to obtain multiple recognition results, wherein each recognition result includes at least one obstacle;

[0038] The obstacles of any data in the second data set are randomly covered by each hook image to obtain a plurality of hook occlusion data, and all the hook occlusion data are added to the second data set to obtain a training data set.

[0039] The present invention takes into account the different types of crane hooks in practical applications. By randomly mixing the acquired images of crane hooks of different models with the images of unobstructed obstacles to generate hook occlusion data, and then constructing a training data set based on the generated hook occlusion data, the diversity of samples in the training data set can be ensured, and the detection model's accurate recognition of each obstacle can be further improved.

[0040] In a second aspect, the present invention provides an obstacle warning system for a blind spot in front of a crane, which is applied to a crane. At least one camera is installed in front of a cab of the crane. The system includes:

[0041] A data acquisition module, used to acquire detection images obtained by each camera detecting the front of the crane;

[0042] The target detection module is used to detect the image features of each detection image, obtain the obstacle features of the target obstacle in each detection image and the obstacle type corresponding to the target obstacle;

[0043] A position determination module, used for extracting a target feature point of a target obstacle from the obstacle features, and determining a target distance between the target obstacle and the crane based on the target feature point;

[0044] The risk warning module is used to determine the target risk level according to the type of target obstacle and the target distance, and control the crane to perform corresponding warning processing based on the target risk level.

[0045] The obstacle warning system for the blind spot in front of the crane of the present invention is applied to a crane with at least one camera installed in front of the cab. By detecting the detection images obtained by each camera in front of the crane, a detection result including the characteristics of different obstacles and their corresponding types is obtained. The distance between the obstacle and the crane is calculated according to the detection result, and the risk level is determined based on the type and distance of the obstacle to control the crane to perform corresponding warning processing. It can overcome the problem of obstacle blind spot recognition and collision detection caused by hook obstruction of the crane, provide a universal and accurate obstacle warning method for the hook obstruction in front of the crane, and greatly improve the operation safety of the crane.

[0046] In a third aspect, the present invention provides a crane, comprising a controller, the controller comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the obstacle warning method in the blind spot in front of the crane according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute a method for warning obstacles in a blind spot in front of a crane according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a flow chart of an obstacle warning method for a blind spot in front of a crane according to an embodiment of the present invention;

[0050] Figure 2 is a flow chart of another method for early warning of obstacles in the blind spot in front of a crane according to an embodiment of the present invention;

[0051] Figure 3 This is a camera installation diagram;

[0052] Figure 4 It is a schematic diagram of the obstacle risk area;

[0053] Figure 5 It is a schematic diagram of the crane blind spot monitoring device;

[0054] Figure 6 is a structural block diagram of an obstacle warning system for a blind spot in front of a crane according to an embodiment of the present invention;

[0055] Figure 7 Schematic diagram of the structure of a controller of a crane according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0057] An embodiment of the present invention provides an obstacle warning method embodiment for the blind spot in front of a crane. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0058] In this embodiment, a method for early warning of obstacles in the blind spot in front of a crane is provided, which is applied to the crane. At least one camera is installed in front of the cab of the crane. Figure 1is a flow chart of an obstacle warning method for a blind spot in front of a crane according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0059] Step S101, acquiring detection images obtained by each camera detecting the front of the crane.

[0060] It should be noted that, in the present embodiment, the camera is used to obtain the environment image in front of the crane, and the specific model and number of the camera are not limited here, and can be adaptively determined according to actual needs.

[0061] Step S102 , detecting image features of each detection image, and obtaining obstacle features of the target obstacle in each detection image and the obstacle type corresponding to the target obstacle.

[0062] In this embodiment, the specific detection method of image features and obstacle types are not limited here, and can be adaptively adjusted according to actual needs. For example, the Scale-Invariant Feature Transform (SIFT) algorithm is used (the main feature is that the algorithm is invariant to changes in image scale, rotation, brightness, etc., and can extract unique and stable feature points, thereby achieving efficient matching and recognition of images); obstacle types include pedestrians, other vehicles, etc., which are only used as examples.

[0063] Step S103: extracting target feature points of the target obstacle from the obstacle features, and determining a target distance between the target obstacle and the crane based on the target feature points.

[0064] In this embodiment, the target feature point represents the corresponding characteristics of the target obstacle, and its specific content is adaptively adjusted according to the obstacle type and feature detection method.

[0065] Step S104, determining a target risk level according to the type of the target obstacle and the target distance, and controlling the crane to perform corresponding warning processing based on the target risk level.

[0066] It should be noted that in this embodiment, the risk level is determined by considering the types of different obstacles and the adaptability of their relative distances to the crane, and the crane is given corresponding early warning processing according to different risk levels; wherein, the specific classification of risk levels and the specific methods of early warning processing can be adaptively adjusted according to actual needs. For example, the risk level includes high risk level and low risk level. If the obstacle is a pedestrian, the collision risk of the pedestrian to the crane is low, and the corresponding target distance value for the high risk level is large; and when the obstacle is other vehicles, since the collision risk of vehicles to the crane is higher than that of pedestrians, the corresponding target distance value for other vehicles of high risk level is small; for the high risk level, the crane can be controlled to perform early warning processing of sound and light alarm, which can only be used as an exemplary description.

[0067] The obstacle warning method for the blind spot in front of the crane in an embodiment of the present invention obtains a detection image in front of the crane through at least one camera installed in front of the crane cab, performs obstacle detection on the detection image, calculates the target distance between the target obstacle and the crane according to the obstacle characteristics of the target obstacle and its corresponding obstacle type, and determines the target risk level based on the type of the target obstacle and the target distance, thereby controlling the crane to perform corresponding warning processing. A universal and effective warning method for obstacles in the blind spot of the crane can be provided, which can overcome the problem of obstacle blind spot identification and collision detection caused by the crane being blocked by the hook, and helps to ensure the safe operation of the crane.

[0068] In this embodiment, a method for early warning of obstacles in the blind spot in front of a crane is provided, which is applied to the crane. At least one camera is installed in front of the cab of the crane. Figure 2 FIG. 1 is a flow chart of another method for early warning of obstacles in the blind spot in front of a crane according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0069] Step S201, obtaining detection images obtained by each camera detecting the front of the crane. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0070] Step S202 , detecting image features of each detection image, and obtaining obstacle features of the target obstacle in each detection image and the obstacle type corresponding to the target obstacle.

[0071] In this embodiment, the trained detection model is used to detect the image features of each detection image, wherein the training process of the detection model includes:

[0072] Step A1, obtaining a training data set, wherein the training data set at least includes hook occlusion data of different obstacles.

[0073] Specifically, the step A1 of obtaining a training data set includes:

[0074] Step A11, respectively collecting first data sets of each camera corresponding to the scene when the crane hangs different types of hooks, and second data sets of each camera corresponding to the scene when the crane does not hang the hook.

[0075] Step A12, extracting a hook image from each data in the first data set to obtain a plurality of hook images.

[0076] In this embodiment, the specific method of extracting the hook image is not limited here, and can be adaptively obtained according to conventional image segmentation algorithms in the field, such as the semantic segmentation algorithm U-Net based on convolutional neural network.

[0077] Step A13: Obstacle identification is performed on each data in the second data set to obtain a plurality of identification results, wherein each identification result includes at least one obstacle.

[0078] In this embodiment, the specific extraction method of obstacle recognition is not limited here, and can be adaptively acquired according to conventional image detection algorithms in the art, such as the SSD (Single Shot MultiBox Detector) target detection algorithm.

[0079] Step A14, using each hook image to randomly cover the obstacles of any data in the second data set to obtain multiple hook occlusion data, and adding all the hook occlusion data to the second data set to obtain a training data set.

[0080] In this embodiment, the hook image is used as a foreground occluder to cover various obstacles at random positions and proportions, thereby obtaining hook occlusion data of different obstacles.

[0081] In the embodiment of the present invention, the different types of crane hooks in actual applications are taken into consideration. By randomly mixing the acquired images of lifting hooks of different models with the images of unobstructed obstacles to generate hook occlusion data, and then constructing a training data set based on the generated hook occlusion data, the diversity of samples in the training data set can be ensured, thereby further improving the detection model's accurate recognition of various obstacles.

[0082] Step A2: training a preset detection model based on a training data set to obtain a trained detection model.

[0083] It should be noted that the specific type of the preset detection model in this embodiment is not limited here, and can be adaptively adjusted according to actual needs. Specifically, this embodiment obtains a trained detection model by training the model using the constructed hook occlusion data containing different obstacles, which can ensure the accurate recognition of image features in each detection image, and further improve the accuracy and efficiency of obstacle warning in the blind spot in front of the crane due to hook occlusion.

[0084] Step S203: extracting target feature points of the target obstacle from the obstacle features, and determining a target distance between the target obstacle and the crane based on the target feature points.

[0085] Specifically, the above step S203 includes:

[0086] Step S2031, calculating the similarity of the detection images detected by each camera.

[0087] In this embodiment, the specific calculation method for detecting the image similarity is not limited here. For example, commonly used similarity indicators such as Euclidean distance, cosine similarity, etc. can be used to analyze the image similarity.

[0088] Step S2032: When the similarity result meets the preset similarity condition, the obstacle features in each detection image are matched to obtain the corresponding target feature points of the target obstacle in each detection image.

[0089] In this embodiment, the specific content of the preset similarity condition is adaptively determined according to the similarity calculation method adopted. For example, the similarity of images is calculated using the Euclidean distance. Since the smaller the Euclidean distance, the greater the similarity, the corresponding preset similarity condition is the set distance threshold, which is only used as an exemplary description.

[0090] Step S2033, calculating the relative distance between each target feature point and the crane, and determining the target distance between the target obstacle and the crane based on the multiple relative distances.

[0091] In this embodiment, the method of determining the target distance through multiple relative distances is not limited here, and is adaptively determined according to actual needs. For example, averaging multiple relative distances and using the average value as the target distance is only for exemplary description.

[0092] In the embodiment of the present invention, similarity calculation is performed on camera detection images obtained in different viewing angles, obstacle feature matching is performed on each detection image when the similarity result meets a preset similarity condition, and then the target feature points of the target obstacle in each detection image are determined according to the matching results, and the target distance between the target obstacle and the crane is determined based on the relative distance between each target feature point and the crane, so that the calculation accuracy of the target distance can be guaranteed, thereby improving the recognition accuracy of the obstacle.

[0093] Step S204, determining a target risk level according to the type of the target obstacle and the target distance, and controlling the crane to perform corresponding warning processing based on the target risk level.

[0094] Specifically, the above step S204 includes:

[0095] Step S2041, determining risk areas of different levels of the target obstacle according to the type of the target obstacle and the preset risk area, wherein the preset risk area marks multiple risk areas of different levels corresponding to different types of obstacles.

[0096] Step S2042: determining a target level risk area from each level risk area of ​​the target obstacle based on the target distance of the target obstacle.

[0097] Step S2043, determining the target risk level of the target obstacle according to the target level risk area, and controlling the crane to perform an audible and visual alarm or a light alarm based on the target risk level.

[0098] In the embodiment of the present invention, the risk areas of different levels corresponding to each obstacle are adaptively defined by the type of each obstacle, the target level risk area of ​​the current obstacle and its target risk level are determined based on the obstacle type actually detected and its distance from the crane, and corresponding sound and light alarms or light alarms are performed based on the target risk level, which can achieve graded processing of collision risks of different vehicles and improve the early warning efficiency of each obstacle in the blind spot in front of the crane.

[0099] In actual applications, due to occasional errors in the determination of obstacle risk levels, that is, the target risk level corresponding to the target obstacle in this embodiment is determined based on the single-frame image data obtained from at least one camera, and there are errors in the recognition of the single-frame image data, so this embodiment also designs a corresponding risk level adjustment process to improve the accuracy of the crane obstacle warning. Therefore, before controlling the crane to perform corresponding warning processing based on the target risk level, the obstacle warning method for the blind spot in front of the crane in this embodiment also includes:

[0100] Step B1, calculating multiple target distances corresponding to the target obstacle within a continuous preset number of frames, and obtaining the relative speed between the target obstacle and the crane corresponding to each frame number.

[0101] In this embodiment, the specific value of the preset number of frames is adaptively adjusted according to actual needs, for example, the preset number of frames is 5 frames.

[0102] Step B2: determining a relative position change trend of the target obstacle based on a plurality of target distances, and determining a relative speed change trend of the target obstacle based on a plurality of relative speeds.

[0103] Step B3: adjusting the target risk level of the target obstacle according to the relative position change trend and the relative speed change trend.

[0104] Specifically, the above step B3 includes:

[0105] Step B31, if the relative position change trend is greater than 0 and the relative speed change trend is greater than 0, the target risk level of the target obstacle is reduced until the level adjustment is stopped when the preset minimum risk level is reached.

[0106] Step B32: If the relative position change trend is greater than 0 and the relative speed change trend is less than 0, or the relative position change trend is less than 0 and the relative speed change trend is greater than 0, the target risk level of the target obstacle is kept unchanged.

[0107] Step B33: if the relative position change trend is less than 0 and the relative speed change trend is greater than 0, the target risk level of the target obstacle is increased until the level adjustment is stopped when the preset highest risk level is reached.

[0108] It should be noted that the specific contents of the preset minimum risk level and the preset maximum risk level in this embodiment can be adaptively set according to actual needs. Specifically, by analyzing the relationship between the relative position change trend and the relative speed change trend of the obstacle and 0, the corresponding risk level adjustment strategy can be determined, which can greatly ensure the accuracy and rationality of the obstacle risk level determination.

[0109] In the embodiment of the present invention, occasional errors in obstacle risk level determination are taken into account, and the detection images obtained through continuous multi-frame detection by the camera are used to calculate multiple target distances corresponding to the same target obstacle and obtain the relative speeds of the target obstacle and the crane corresponding to the corresponding number of frames. The relative position change trend and the relative speed change trend of the obstacle are determined according to the multiple target distances and relative speeds, and the target risk level of the obstacle is adjusted based on the relative position change trend and the relative speed change trend. This can improve the determination accuracy of the obstacle risk level and further ensure the accuracy and effectiveness of obstacle risk warning.

[0110] In a specific embodiment, a crane front blind spot monitoring and collision warning solution is provided to achieve blind spot monitoring and collision object identification during the crane's travel with a hook, to warn of collision risks, and to improve the safety of crane travel. Specifically, the specific process of the above solution includes:

[0111] 1. Install the camera.

[0112] In this embodiment, see Figure 3A camera (i.e., an ordinary camera) is installed on each side of the left and right sides in front of the crane's cab. The two cameras are used to fill in each other's blind spots. The camera installation position only needs to cover the blind spot blocked by the hook as much as possible, that is, the camera installation position can reduce the obstruction of the field of vision by the front hook.

[0113] 2. Delineate risk areas.

[0114] In this embodiment, different risk areas are defined for obstacles of four types of dynamic obstacles, including pedestrians, cyclists, small cars and large cars. For details, see Figure 4 The first-level risk area is 0-d(a / b / c / d)1, the second-level risk area is d(a / b / c / d)1-d(a / b / c / d)2, and the third-level risk area is d(a / b / c / d)2-d(a / b / c / d)3; among them, a / b / c / d represent four types of dynamic obstacles, such as the first-level risk area for cyclists is 0-db1, the second-level risk area is db1-db2, and the third-level risk area is db3-db4, which is only for illustrative purposes.

[0115] 3. Construct a blind spot monitoring device in front of the crane.

[0116] In this embodiment, Figure 5 This is a schematic diagram of a crane blind spot monitoring device. Figure 5 It can be seen that the device includes two stages, namely, the I training stage and the II application stage; the device is specifically composed of a target ranging module, a target detection module, a data generation module and a decision module. The workflow of each module in different stages is shown below.

[0117] In the training phase, in order to quickly generate a hook occlusion dataset of various obstacles, the data generation module adopts a method of mixing and synthesizing unobstructed obstacle images with hook images at random positions. Specifically, firstly, a dataset A of different types of hooks in the perspective of two cameras in front of the cab is collected, and a dataset B of different obstacles in the perspective of two cameras in front of the cab in the no-hook scenario is collected. The hooks in dataset A are cropped out as foreground occluders, and the obstacles in dataset B are covered at random positions and proportions to generate a synthetic dataset C. The synthetic dataset C is used to train the target detection model in the target detection module.

[0118] In the application stage, the device is integrated into the crane, and the target detection module, target ranging module and decision module are run in the vehicle controller (i.e., the main controller of the crane). Specifically, the real-time data of the two cameras in front are first input into the target detection module, and two target detection models (such as yolov5) are used to detect the target of the two camera data respectively, and the detected obstacle type is output to the decision module, that is, the image in the envelope of the detected obstacle is output to the target ranging module; then, the target ranging module extracts and matches the image in the envelope output by the two different camera data, and calculates the relative position of the target obstacle and the crane based on the binocular stereo vision measurement technology for the successfully matched feature points, and outputs the obstacle position information to the decision module; finally, the decision module outputs different levels of warning information according to the type and position of the obstacle, such as giving a high-level alarm signal (such as sound and light alarm) for vulnerable traffic participants such as pedestrians and bicycles at close range, and giving a low-level alarm signal (such as light alarm) for vulnerable traffic participants or vehicles at a longer distance.

[0119] In this embodiment, a specific implementation of a crane front blind spot monitoring and collision warning solution is provided, including:

[0120] 1. Obtain the occlusion mixed data set and training model. The specific process includes:

[0121] Step 1: Collect image data of different types of hooks hanging in front of the crane in driving state, and use image segmentation algorithm (such as U-Net, SegNet) to extract the hook image.

[0122] Step 2: Collect image data of the crane in a driving state with no hook hanging in front and different types of obstacles (such as pedestrians, bicycles, vehicles, etc.), and use the target detection algorithm (such as yolov5) to obtain the obstacle envelope.

[0123] Step 3: According to the position and size of the envelope frame, the hook image obtained in the first step is used as the foreground and mixed with the obstacle data collected in the second step, so that the hook image occludes obstacles at random positions and proportions in the vicinity of the envelope frame, generating a synthetic data set with different obstacle occlusion conditions.

[0124] Step 4: Mix the image dataset collected in the second step with the synthetic dataset obtained in the third step to train the target detection model (such as YOLOv).

[0125] 2. Detect obstacle targets, extract target image feature points, and calculate target positions based on the feature points of two cameras. The specific process includes:

[0126] Step 1: transplant the trained target detection model into the crane controller, and use a target detection model to perform target detection on the image data collected by the two cameras. It should be noted that this embodiment can also use a target detection model, which needs to be trained with the image data of the two cameras.

[0127] Step 2: Output the obstacle types detected in the two-channel image data to the decision module, and output the image within the envelope of the detected obstacle to the target ranging module.

[0128] Step 3: Perform feature extraction (such as SIFT algorithm, Harris algorithm, etc.) on the image within the envelope frame of the target detection results of the image data of the two cameras.

[0129] Step 4: Compare the similarity of the feature descriptors of the two image data obtained in the first step (such as using Euclidean distance to measure the similarity). When the similarity is greater than the set threshold, the pixels in the two image data can be matched to obtain a set of matching pixels.

[0130] Step 5: Based on binocular stereo vision measurement technology, the position of the matching feature points relative to the crane is calculated based on the matching pixel point set to form an obstacle feature point cloud, and the center of gravity position of the obstacle feature point cloud is used as the obstacle position.

[0131] 3. Alarm decision-making, the specific process includes:

[0132] Step 1: Use the relative position between the obstacle and the crane and the obstacle type to establish a risk assessment model and score the obstacle collision risk.

[0133] Step 2: Calculate the relative position change trend between the obstacle and the crane in the last few consecutive frames (such as 8 frames). If they are relatively close, increase the risk level score; if they are relatively far away, reduce the risk level score.

[0134] Step 3: Output different forms of alarms based on the risk level score, such as sound and light alarms for high risk levels and only light alarms for low risk levels.

[0135] It should be noted that, see Figure 4 In this embodiment, four types of obstacles are included, namely pedestrians, cyclists, small vehicles and large vehicles. Different initial risk levels are assigned according to the risk area where the obstacle is located when the obstacle is first detected. After that, the relative distance change Δd between the same obstacle and the crane in the driving direction and the average speed v of the obstacle relative to the vehicle in consecutive frames are detected. The risk level is determined according to the relative distance change Δd and the average speed v (i.e., the risk level is jointly determined by the alarm range, the relative distance change Δd between the obstacle and the vehicle and the average speed v of the obstacle relative to the vehicle), specifically:

[0136] 1) If Δd>0 and v>0 (i.e. the obstacle is relatively far away and will continue to move away), the risk level is reduced to the lowest level of risk based on the risk level of the area where the obstacle is currently located.

[0137] 2) If Δd>0 and v<0 (i.e., the obstacle is relatively far away but will get closer later) or Δd<0 and v>0 (i.e., the obstacle is relatively close but will get farther away later), maintain the risk level of the area where the obstacle is currently located.

[0138] 3) If Δd<0 and v<0 (the obstacle is relatively close and will continue to get closer), the risk level is increased based on the risk level of the area where the obstacle is currently located to the highest level of risk.

[0139] In one specific embodiment, see Figure 4 , it is known that the first-level risk area for cyclists is 0-db1, the second-level risk area is db1-db2, and the third-level risk area is db3-db4. Specifically, when a cyclist crosses the second-level risk area, in theory, both Δd and v are 0, and the risk level of the area remains unchanged. At this time, the crane will continue to issue a second-level alarm; however, when the cyclist is traveling in the same direction as the vehicle in the second-level risk area, if Δd<0 and v<0, even if the area where the cyclist is located is a second-level risk area, the risk level will be upgraded to level one; if Δd>0 and v>0, even if the area where the cyclist is located is a second-level risk area, the risk level will be downgraded to level three.

[0140] It should be noted that the alarm process under different collision warning situations in this embodiment is as follows:

[0141] 1. Controller: Run two target detection modules to calculate and process the video information received from the two cameras respectively, and obtain the envelope of the target at two different perspectives, and then process it based on the early warning process in the above-mentioned specific implementation method, that is, the controller uses the above-mentioned early warning process to calculate the relative position of the obstacle and the vehicle; then, according to the relative position of the obstacle and the vehicle, and combined with the status information of the vehicle, an alarm signal is output according to the alarm decision plan in the specific implementation method.

[0142] 2. Instrument: Receives alarm signals from the controller and generates light alarm for low-level warning information and sound and light alarm for high-level alarm information.

[0143] It should be noted that when the obstacle in front of the crane is small and close to the hook, the obstacle will inevitably be obscured to a certain extent. The above-mentioned early warning process of this embodiment, that is, the target detection method based on binocular stereo vision can still perform efficient detection of the distance and type of the obstacle obscured by the hook.

[0144] In summary, the obstacle warning method for the blind spot in front of the crane according to the embodiment of the present invention can realize the accurate detection of various obstacles in the blind spot in front caused by the obstruction of the hook in front of the crane and the effective warning of the collision risk, which greatly ensures the safe operation of the crane.

[0145] In this embodiment, an obstacle warning system for the blind spot in front of the crane is also provided. The system is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are not repeated here. As used below, the term "module" refers to a combination of software and / or hardware that can implement a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0146] The present invention provides an obstacle warning system for the blind spot in front of a crane, which is applied to the crane. At least one camera is installed in front of the crane cab. Figure 6 As shown, the system includes:

[0147] The data acquisition module 601 is used to acquire the detection images obtained by each camera detecting the front of the crane.

[0148] The target detection module 602 is used to detect image features of each detection image, and obtain obstacle features of the target obstacle in each detection image and the obstacle type corresponding to the target obstacle.

[0149] The position determination module 603 is used to extract target feature points of the target obstacle from the obstacle features, and determine the target distance between the target obstacle and the crane based on the target feature points.

[0150] The risk warning module 604 is used to determine the target risk level according to the type of the target obstacle and the target distance, and control the crane to perform corresponding warning processing based on the target risk level.

[0151] In some optional embodiments, the target detection module 602 includes: a first detection submodule and a second detection submodule; wherein the first detection submodule is used to obtain a training data set, and the training data set at least contains hook occlusion data of different obstacles; the second detection submodule is used to train a preset detection model based on the training data set to obtain a trained detection model.

[0152] In some optional embodiments, the first detection submodule includes: a first acquisition unit, a second acquisition unit, a third acquisition unit and a fourth acquisition unit; wherein the first acquisition unit is used to respectively collect the first data sets of each camera corresponding to the crane in the scene of hanging different types of hooks, and the second data set of each camera corresponding to the crane in the scene of not hanging the hook; the second acquisition unit is used to extract the hook image for each data in the first data set to obtain multiple hook images; the third acquisition unit is used to perform obstacle identification for each data in the second data set to obtain multiple identification results, wherein each identification result includes at least one obstacle; the fourth acquisition unit is used to use each hook image to randomly cover the obstacles of any data in the second data set to obtain multiple hook occlusion data, and add all hook occlusion data to the second data set to obtain a training data set.

[0153] In some optional embodiments, the position determination module 603 includes: a first determination submodule, a second determination submodule and a third determination submodule; wherein the first determination submodule is used to calculate the similarity of the detection images detected by each camera; the second determination submodule is used to match the obstacle features in each detection image when the similarity result meets the preset similarity condition, and obtain the target feature points of the target obstacle in each detection image; the third determination submodule is used to calculate the relative distance between each target feature point and the crane, and determine the target distance between the target obstacle and the crane based on multiple relative distances.

[0154] In some optional embodiments, the risk warning module 604 includes: a first warning sub-module, a second warning sub-module and a third warning sub-module; wherein the first warning sub-module is used to determine various levels of risk areas of the target obstacle according to the type of the target obstacle and the preset risk area, and the preset risk area marks multiple levels of risk areas corresponding to different types of obstacles; the second warning sub-module is used to determine the target level risk area from the various levels of risk areas of the target obstacle based on the target distance of the target obstacle; the third warning sub-module is used to determine the target risk level of the target obstacle according to the target level risk area, and control the crane to perform sound and light alarm or light alarm based on the target risk level.

[0155] In some optional embodiments, the system also includes: a level adjustment module, which is used to calculate multiple target distances corresponding to the target obstacle within a continuous preset number of frames, and obtain the relative speed between the target obstacle and the crane corresponding to each frame number; determine the relative position change trend of the target obstacle based on the multiple target distances, and determine the relative speed change trend of the target obstacle based on the multiple relative speeds; and adjust the target risk level of the target obstacle according to the relative position change trend and the relative speed change trend.

[0156] In some optional embodiments, the level adjustment module includes: an adjustment submodule, used to reduce the target risk level of the target obstacle if the relative position change trend is greater than 0 and the relative speed change trend is greater than 0, and stop the level adjustment until the preset minimum level risk is reached; if the relative position change trend is greater than 0 and the relative speed change trend is less than 0, or the relative position change trend is less than 0 and the relative speed change trend is greater than 0, keep the target risk level of the target obstacle unchanged; if the relative position change trend is less than 0 and the relative speed change trend is greater than 0, increase the target risk level of the target obstacle, and stop the level adjustment until the preset maximum level risk is reached.

[0157] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0158] The obstacle warning system for the blind spot in front of the crane in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0159] The obstacle warning system for the blind spot in front of the crane in the embodiment of the present invention can realize accurate detection of obstacles in the blind spot in front of the crane due to obstruction by the hook in front of the crane and effective warning of collision risks, thereby ensuring the safe operation of the crane to a great extent.

[0160] The present invention also provides a crane in an embodiment, the crane includes a controller, see Figure 7 , Figure 7 is a schematic diagram of the structure of the controller provided in an optional embodiment of the present invention, such as Figure 7 As shown, the controller includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Each component utilizes different buses to communicate with each other, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the master controller, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple master controllers can be connected, and each master controller provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0161] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0162] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0163] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the controller, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the controller via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0164] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0165] The controller also includes a communication interface 30 for the main control chip to communicate with other devices or a communication network.

[0166] A computer-readable storage medium is also provided in an embodiment of the present invention. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor main control chip or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0167] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. An obstacle warning method for a blind spot in front of a crane, applied to a crane, wherein at least one camera is installed in front of the cab of the crane, characterized in that: The method comprises: Acquire detection images obtained by each camera detecting the front of the crane; Detecting image features of each of the detection images to obtain obstacle features of the target obstacles in each of the detection images and obstacle types corresponding to the target obstacles; extracting a target feature point of the target obstacle from the obstacle features, and determining a target distance between the target obstacle and the crane based on the target feature point; A target risk level is determined according to the type of the target obstacle and the target distance, and the crane is controlled to perform corresponding early warning processing based on the target risk level.

2. The obstacle warning method for the blind spot in front of the crane according to claim 1 is characterized in that: The step of extracting a target feature point of the target obstacle from the obstacle feature and determining a target distance between the target obstacle and the crane based on the target feature point comprises: Calculating similarity of the detection images detected by each of the cameras; When the similarity result meets the preset similarity condition, the obstacle features in each of the detection images are matched to obtain the corresponding target feature points of the target obstacle in each of the detection images; The relative distance between each of the target feature points and the crane is calculated, and the target distance between the target obstacle and the crane is determined based on the multiple relative distances.

3. The obstacle warning method for the blind spot in front of the crane according to claim 1 is characterized in that: Determining a target risk level according to the type of the target obstacle and the target distance, and controlling the crane to perform corresponding early warning processing based on the target risk level, includes: Determine risk areas of different levels of the target obstacle according to the type of the target obstacle and the preset risk area, wherein the preset risk area marks multiple risk areas of different levels corresponding to different types of obstacles; Determining a target level risk area from each level risk area of ​​the target obstacle based on a target distance of the target obstacle; The target risk level of the target obstacle is determined according to the target risk level risk area, and the crane is controlled to perform an audible and visual alarm or a light alarm based on the target risk level.

4. The obstacle warning method for the blind spot in front of the crane according to claim 1 is characterized in that: Before controlling the crane to perform corresponding early warning processing based on the target risk level, the method further includes: Calculating multiple target distances corresponding to the target obstacle within a continuous preset number of frames, and obtaining a relative speed between the target obstacle and the crane corresponding to each frame number; Determine a relative position change trend of the target obstacle based on a plurality of target distances, and determine a relative speed change trend of the target obstacle based on a plurality of relative speeds; The target risk level of the target obstacle is adjusted according to the relative position change trend and the relative speed change trend.

5. The obstacle warning method for the blind spot in front of the crane according to claim 4 is characterized in that: The step of adjusting the target risk level of the target obstacle according to the relative position change trend and the relative speed change trend includes: If the relative position change trend is greater than 0 and the relative speed change trend is greater than 0, the target risk level of the target obstacle is reduced until the level adjustment is stopped when the preset minimum risk level is reached; If the relative position change trend is greater than 0 and the relative speed change trend is less than 0, or the relative position change trend is less than 0 and the relative speed change trend is greater than 0, the target risk level of the target obstacle remains unchanged; If the relative position change trend is less than 0 and the relative speed change trend is greater than 0, the target risk level of the target obstacle is increased until the level adjustment is stopped when the preset highest risk level is reached.

6. The obstacle warning method for the blind spot in front of the crane according to claim 1 is characterized in that: The trained detection model is used to detect the image features of each of the detection images, wherein the training process of the detection model includes: Acquire a training data set, wherein the training data set at least includes hook occlusion data of different obstacles; The preset detection model is trained based on the training data set to obtain a trained detection model.

7. The obstacle warning method for the blind spot in front of the crane according to claim 6 is characterized in that: The step of obtaining a training data set includes: Collecting first data sets of each camera corresponding to a scene in which the crane is hung with hooks of different models, and second data sets of each camera corresponding to a scene in which the crane is not hung with hooks; Extracting a hook image from each data in the first data set to obtain a plurality of hook images; Perform obstacle identification on each data in the second data set to obtain multiple identification results, wherein each identification result includes at least one obstacle; Obstacles of any data in the second data set are randomly covered by each hook image to obtain a plurality of hook occlusion data, and all hook occlusion data are added to the second data set to obtain a training data set.

8. An obstacle warning system for the blind spot in front of a crane, applied to a crane, wherein at least one camera is installed in front of the cab of the crane, characterized in that: The system comprises: A data acquisition module, used to acquire detection images obtained by each camera detecting the front of the crane; A target detection module, used to detect image features of each of the detection images, and obtain obstacle features of target obstacles in each of the detection images and obstacle types corresponding to the target obstacles; a position determination module, configured to extract a target feature point of the target obstacle from the obstacle feature, and determine a target distance between the target obstacle and the crane based on the target feature point; The risk warning module is used to determine the target risk level according to the type of the target obstacle and the target distance, and control the crane to perform corresponding warning processing based on the target risk level.

9. A crane, characterized in that: The crane includes a controller, and the controller includes: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the obstacle warning method for the blind spot in front of the crane according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the obstacle warning method for the blind spot in front of a crane according to any one of claims 1 to 7.

Images