Identification positioning method applied to engineering machinery and engineering machinery

By combining image detection and lidar technology, data is integrated to improve pedestrian positioning accuracy, the problem of inaccurate pedestrian positions in engineering machinery operations is solved, and a safer operating environment is achieved.

CN118915084APending Publication Date: 2024-11-08SANY MARINE HEAVY INDUSTRY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410851697.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Construction machinery has a safety risk of impacting pedestrians during operation. The existing image detection method based on image detection is inaccurate in camera distortion, inaccurate calibration and light influence, which cannot effectively reduce safety hazards.

Method used

Combining image detection and lidar technology, pedestrian location is determined through image detection, and lidar is used for accurate positioning, data is fused to improve pedestrian positioning accuracy, target location is determined, and control strategies are determined based on position differences and distances to reduce safety risks.

Benefits of technology

It improves the positioning accuracy of pedestrian positions in the construction machinery operation area, reduces safety hazards, and ensures more accurate safety monitoring and risk prevention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915084A_ABST
    Figure CN118915084A_ABST
Patent Text Reader

Abstract

The invention provides a recognition and positioning method applied to engineering machinery, and the method comprises the steps: carrying out the image detection of an operation region of the engineering machinery, and determining a first position where a pedestrian is located in the operation region; performing laser radar detection on the operation area, and determining a second position of an obstacle in the operation area; and determining the target position of the pedestrian according to the first position and the second position, thereby improving the positioning precision of the target position of the pedestrian.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of engineering machinery, and in particular to an identification and positioning method applied to engineering machinery and the engineering machinery. Background Art

[0002] Construction machinery (such as front loaders, etc.) has a safety risk of hitting pedestrians during normal operation. The relevant technology provides a method based on image detection, for example, a 360-degree panoramic imaging system. However, this system transmits the 360-degree image to the cab, and the driver manually identifies the position of the pedestrian. Moreover, the position of the pedestrian obtained by this method may not be accurate enough due to camera distortion, inaccurate camera calibration, or changes in the pedestrian's posture, light, and shadow. Since the driver cannot obtain the exact position of the pedestrian, there is still a safety risk in the operation of the construction machinery. Summary of the invention

[0003] In view of this, the embodiments of the present application are directed to providing an identification and positioning method for engineering machinery and engineering machinery.

[0004] In a first aspect, a recognition and positioning method for construction machinery is provided, the method comprising: performing image detection on the working area of ​​the construction machinery to determine a first position of pedestrians in the working area; performing laser radar detection on the working area to determine a second position of obstacles in the working area; and determining a target position of the pedestrian based on the first position and the second position.

[0005] In one embodiment, a method for determining a target position of a pedestrian through a first position and a second position includes: if the difference between the first position and the second position satisfies a first preset condition, determining the second position as the target position; and / or if the difference between the first position and the second position does not satisfy the first preset condition, determining that no pedestrian appears in the working area.

[0006] In one embodiment, image detection is performed on the working area of ​​the engineering machinery to determine the first position of the pedestrian in the working area, including: performing image detection on the working area to determine the third position of the pedestrian, the third position being the position of the pedestrian in the image coordinate system; converting the third position into the first position, the first position being the position of the pedestrian in the world coordinate system where the engineering machinery is located.

[0007] In one embodiment, a laser radar detection is performed on the working area to determine a second position of an obstacle in the working area, including: performing a laser radar detection on the working area to determine a fourth position of the obstacle, the fourth position being the position of the obstacle in the laser radar coordinate system; converting the fourth position into the second position, the second position being the position of the obstacle in the world coordinate system where the engineering machinery is located.

[0008] In one embodiment, a laser radar is used to track the position of a pedestrian, including: determining a target position before the pedestrian disappears from a detection range; detecting the position of an obstacle closest to the target position using the laser radar; and updating the target position to the obstacle position if the difference between the obstacle position and the target position meets a second preset condition.

[0009] In one embodiment, determining the control strategy of the engineering machine according to the distance between the target position and the engineering machine includes: determining the control strategy of the engineering machine according to the distance between the target position and the engineering machine and the moving direction of the pedestrian.

[0010] In one embodiment, the control strategy includes one or more of the following: issuing an alarm, limiting the action of the actuator of the engineering machinery, and limiting the moving speed of the engineering machinery.

[0011] In the second aspect, a construction machinery includes: an image detection system for performing image detection on the working area of ​​the construction machinery; a laser radar system for performing laser radar detection on the working area; and a processing system for executing the identification and positioning method in the above embodiment based on data collected by the image detection system and the laser radar system.

[0012] The first position mentioned above is obtained based on image detection, and the second position is obtained based on laser radar. It can be seen that the embodiment of the present application fuses image detection data and laser radar data, which helps to improve the positioning accuracy of pedestrians in the operation area of ​​engineering machinery, thereby reducing safety hazards during the operation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of the identification and positioning method provided in an embodiment of the present application.

[0014] Figure 2 A schematic diagram of a pedestrian positioning method provided in an embodiment of the present application.

[0015] Figure 3 A schematic diagram of a LiDAR deployment method provided in an embodiment of the present application.

[0016] Figure 4A A schematic diagram of the position of pedestrians detected by the image detection module provided in an embodiment of the present application.

[0017] Figure 4B A schematic diagram of the position of pedestrians detected by the laser radar provided in an embodiment of the present application.

[0018] Figure 5 A schematic diagram of a process for obtaining a pedestrian's position provided in an embodiment of the present application.

[0019] Fig. 6A FIG. 5 is a flow chart of a possible implementation of step S510 .

[0020] Figure 6B A flowchart of a possible implementation of step S520 is shown in FIG.

[0021] Figure 7 A schematic diagram of an image transformation method provided in an embodiment of the present application.

[0022] Figure 8 A schematic diagram of a coordinate transformation method provided in an embodiment of the present application.

[0023] Fig. 9 A schematic diagram of a real-time detection transformer (RT-DETR) algorithm model architecture provided in an embodiment of the present application.

[0024] Fig.10 A schematic diagram of another process of obtaining the position of a pedestrian provided in an embodiment of the present application.

[0025] Fig.11A A schematic diagram of a possible implementation of step S910.

[0026] Fig. 11B A schematic diagram of a possible implementation of step S920.

[0027] Fig.12 A schematic diagram of a pedestrian tracking process proposed in an embodiment of the present application.

[0028] Fig.13 This is a schematic diagram of another process of pedestrian tracking proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0030] Construction machinery generally refers to various mechanical equipment used in the fields of construction, civil engineering, transportation, water conservancy, mining, etc., which are used for earthwork, excavation, transportation, lifting, compaction, concrete construction, road paving and other operations; the movement of construction machinery plays an important role in various engineering operations, such as improving construction efficiency, realizing multi-functional operations, increasing operating height and range, and improving work safety, etc.

[0031] At present, common construction machinery, such as front loaders, will have the safety risk of hitting pedestrians during normal construction operations. In the related technology, a method based on image detection is proposed to improve the safety of construction machinery equipment in normal construction operations by determining the position of pedestrians.

[0032] For example, a 360-degree panoramic imaging system can be used on construction machinery to determine the location of pedestrians. The 360-degree panoramic imaging system is an on-board image safety system that collects images around the vehicle body by installing several wide-angle cameras around the vehicle body, and through image processing, the images taken by these multiple cameras can be spliced ​​into an overall image of the vehicle body for display to the driver. This can reduce traffic accidents caused by the driver's blind spots. In addition, the 360-degree panoramic imaging system can transmit images around the vehicle body to the cab in real time, making it convenient for the driver to grasp the environmental characteristics of the close range in real time and find pedestrians in time to avoid accidents.

[0033] However, although the driver can obtain real-time images around the vehicle through the 360-degree panoramic imaging system, he can only determine the position of pedestrians based on the image. However, due to camera distortion, inaccurate camera calibration, or changes in the pedestrian's posture, light and shadow, the position of the pedestrian may not be accurate enough. Since the driver cannot obtain the accurate position of the pedestrian, there is still a safety risk in the operation of the construction machinery.

[0034] Compared with the image detection positioning method mentioned above, the positioning method based on laser radar has higher positioning accuracy. However, the positioning method based on laser radar can only identify obstacles, but cannot identify the type of obstacles (that is, it cannot determine whether the obstacles are pedestrians).

[0035] The embodiment of the present application combines the positioning method based on image detection and the positioning method based on laser radar. The positioning method based on image detection can be mainly used to identify pedestrians, and the positioning method based on laser radar can be mainly used to accurately locate pedestrians. It can be seen that the embodiment of the present application can simultaneously utilize the advantages of image detection and laser radar, thereby improving the accuracy of pedestrian positioning.

[0036] Combine the following Figure 1 , the embodiments of the present application are introduced in detail.

[0037] Figure 1 1 is a flow chart of a method for identifying and locating an engineering machine provided in an embodiment of the present application. Figure 1 As shown, the identification and positioning method includes the following steps:

[0038] Step S110: performing image detection on the working area of ​​the construction machinery, and determining a first position of a pedestrian in the working area through image detection.

[0039] The construction machinery mentioned in the embodiments of the present application may be any type of mechanical equipment used in construction operations. For example, the construction machinery may be a large-scale machine such as a reach stacker, a forklift or a heavy forklift. The operation area of ​​the construction machinery may be determined based on the range of motion of the actuator of the construction machinery.

[0040] The first position may be a position determined by image detection. For example, the first position may be represented by coordinates. The coordinates may be coordinates in an image coordinate system or coordinates in a world coordinate system. Figure 2 , the first position may indicate the coordinate position (x1, y1) of the pedestrian in the vehicle coordinate system (the world coordinate system centered on the vehicle).

[0041] The image detection mentioned in step S110 can be performed by a camera installed on the construction machinery. The embodiment of the present application does not specifically limit the installation method of the camera on the construction machinery. In some embodiments, four cameras can be installed on the construction machinery to realize a 360-degree all-round picture detection function around the construction machinery. These four cameras can cover the four directions of the construction machinery respectively, such as: front, rear, left and right, and these four directions can also be the left front, left rear, right front, and right rear positions of the construction machinery. This embodiment does not limit the specific position of the camera. These four cameras can capture images of the surrounding environment in real time. By integrating the images of the four cameras, all-round monitoring of the situation around the vehicle can be achieved. By performing image processing and analysis on these images, pedestrians are identified, and by comparing and matching the images of the four cameras, the first position of the pedestrian based on the construction machinery is calculated.

[0042] Step S120: a laser radar detection may be performed on the working area to determine a second position of obstacles in the working area.

[0043] LiDAR detection can accurately detect surrounding obstacles, including the exact position of pedestrians, by acquiring three-dimensional spatial information of the surrounding environment, including distance, height and other data.

[0044] Based on the working area of ​​the construction machinery, that is, the range of image detection, laser radar detection can be used to scan in the working area, and the position of the obstacle detected can be the second position. The second position can be the position of the obstacle in the laser radar coordinate system or the position of the obstacle in the world coordinate system. Taking the construction machinery as a front loader as an example, the second position can indicate the coordinate position of the obstacle in the vehicle coordinate system (the world coordinate system centered on the vehicle).

[0045] The embodiments of the present application do not specifically limit the arrangement of the laser radar sensors on the construction machinery. For example, multiple laser radar sensors can be arranged according to the scanning direction of the laser radar sensor, so that the laser radar can fully scan the working area of ​​the construction machinery. As an example, two laser radar sensors can be installed on the construction machinery, and the coverage range of each laser radar sensor is 270 degrees, so that a panoramic 360-degree coverage of the construction machinery can be achieved. Figure 3 As shown, the two laser radar sensors can be installed on the left front side and the right rear side of the engineering machine, or on the right front side and the left rear side of the engineering machine, respectively, to achieve 360-degree full coverage.

[0046] Step S130: Determine the target position of the pedestrian based on the first position and the second position. For example, based on the first position and the second position, it can be confirmed whether the obstacle at the second position scanned by the laser radar is the pedestrian identified in the image detection. If the pedestrian is identified, the second position can be further determined as the target position of the pedestrian. Alternatively, the position between the first position and the second position can also be determined as the target position of the pedestrian.

[0047] Because there may be errors in the detection of pedestrian coordinates during the image detection process. The errors may be caused by a variety of factors, including camera distortion, different perspectives and posture changes of pedestrians, occlusion problems, lighting and shadow effects, complex background interference, etc. Therefore, there will be some deviation between the first position of the pedestrian obtained by image detection and the target position of the pedestrian. In this application, by integrating the data of image detection and lidar detection, the positioning accuracy of the target position of pedestrians is improved, ensuring the accurate identification of the position of pedestrians in the operation area of ​​engineering machinery, thereby providing a more accurate basis for safety monitoring and risk prevention.

[0048] In some embodiments, the target position may be confirmed by confirming whether the difference between the first position and the second position satisfies a first preset condition.

[0049] Optionally, when the difference satisfies a first preset condition, the second position can be determined as the target position.

[0050] The difference between the first position and the second position can be understood as the straight-line distance between the first position and the second position. The straight-line distance can be based on the distance between the two positions in a two-dimensional plane or the distance between the two positions in a three-dimensional space.

[0051] In the related art, a variety of calculation methods can be used to calculate the difference between two locations. The selection of the calculation method can be based on the characteristics of the processed data, the requirements of the application scenario, and the expected distance measurement properties.

[0052] Below, the solution of the present application is described by taking the calculation of the difference between two positions by using the norm as an example.

[0053] First, assume that the first position and the second position are both in a two-dimensional plane. Figure 4A As shown, X is the first position of the pedestrian in the operation area determined by image detection; Figure 4B As shown, P is the second position of the pedestrian in the working area determined by laser radar detection.

[0054] Secondly, the difference between the first position and the second position can be calculated by the norm.

[0055] The norm is a mathematical concept that measures the size of a vector and can be used to calculate the distance between two locations. The coordinates X of the first location and the coordinates P of the second location can be expressed in vector form. For example, in this embodiment, X and P are points in a two-dimensional space, that is, X = (x1, y1), P = (x2, y2). Then the vector difference between X and P can be expressed as

[0056] X―P=(x1―x2, y1―y2)

[0057] Alternatively, the difference between the first position and the second position may be expressed as norm(X-P).

[0058] In some embodiments, the norm calculation can be performed using different functions, such as Manhattan distance calculation.

[0059] norm(X―P)=|x1―x2|+|y1―y2|

[0060] Alternatively, the Euclidean distance can be used,

[0061]

[0062] In the embodiments of the present application, there is no restriction on the specific function content of the norm, and only the calculated difference value is concerned.

[0063] Finally, the difference is compared with the first preset condition to determine whether the difference meets the first preset condition.

[0064] The first preset condition can be understood as a condition for determining whether the obstacle scanned by the laser radar is a pedestrian. The determination of the type of the first preset condition depends on the selection of the above-mentioned difference calculation method.

[0065] For example, when the norm is used to calculate the position difference, the first preset condition may be a specific value, such as 1, 2, etc. Alternatively, the first preset condition may also be a range.

[0066] When the above difference is smaller than the first preset condition, we can consider that the obstacle scanned by the laser radar is the pedestrian detected by the image.

[0067] For example, when the first preset condition is 1, that is, norm(X-P)<1, it can be considered that the obstacle detected by the laser is a pedestrian.

[0068] It can be further understood that after a pedestrian is identified through image detection and the first position of the pedestrian is obtained, the area is scanned by the laser radar to confirm the second position of the obstacle, and then the first position and the second position are detected and confirmed. When the difference between the two positions meets the first preset condition, it can be confirmed that the obstacle detected at the second position is a pedestrian.

[0069] Since laser radar is not affected by changes in light and has strong anti-interference capabilities, the detection accuracy based on laser radar is better than that of image detection. Therefore, when it is confirmed that the obstacle at the second position is the pedestrian at the first position, the second position can be determined as the target position of the pedestrian. Compared with the first position based on image detection, the target position has higher accuracy.

[0070] Optionally, if the difference between the first position and the second position does not satisfy the first preset condition, it can be determined that no pedestrian appears in the working area.

[0071] Taking the above embodiment as an example, when norm(X―P)<1, that is, the difference between the first position and the second position satisfies the first preset condition, it can be considered that the obstacle detected by the laser is a pedestrian. When norm(X―P)>1, that is, the difference between the first position and the second position does not satisfy the first preset condition, it can be considered that the obstacle detected by the laser radar is not a pedestrian in the image detection.

[0072] In the present application, whether a pedestrian appears in the operation area can be further determined by whether the difference between the first position and the second position satisfies the first preset condition. The target position of the pedestrian in the operation area of ​​the construction machinery is determined by combining the first position detected by the image with the second position detected by the laser radar and by the difference between the first position and the second position. This further improves the accuracy of the pedestrian position. This allows the driver of the construction machinery to better identify the position of the pedestrian and avoid safety hazards.

[0073] In some embodiments, image detection is performed on the working area of ​​the engineering machinery, and the first position of the pedestrian in the working area can be determined by the following steps.

[0074] Figure 5 It is a schematic diagram of a process of obtaining a first position through image detection provided in an embodiment of the present application.

[0075] Step S510: performing image detection on the working area to determine a third position of the pedestrian, where the third position is the position of the pedestrian in the image coordinate system.

[0076] First, the present application can use cameras installed around the body of the construction machinery to realize image monitoring of its working area. As described in the above embodiment, four cameras can be installed on the construction machinery to realize the 360-degree all-round image detection function around the construction machinery. When the camera starts working, the four cameras will simultaneously monitor the area they cover. When a pedestrian appears in the area monitored by the image, it can be understood that the position of the pedestrian relative to the camera, that is, the position of the pedestrian in the image coordinate system formed by the camera, is the third position. Fig. 6A The process is described as an example.

[0077] Step S610: When a pedestrian is detected in the image, an identification frame of the pedestrian can be obtained.

[0078] During the image detection process, each camera can collect image data of pedestrians in the camera area. Pedestrians in these images can be annotated, and the boundary box of the pedestrian can be drawn based on the annotation. It can also be understood as the identification box of the pedestrian.

[0079] Since the images are taken by different cameras, it can be understood that the position of the pedestrian's identification frame is determined based on the position of the camera.

[0080] Step S620: The bottom of the identification frame is used as the ground position coordinates where the pedestrian is standing, that is, the third position of the pedestrian is obtained.

[0081] Based on the above image, a recognition frame of the pedestrian can be identified. In the image, an image coordinate system is formed with the camera as the origin, and the third position of the pedestrian based on the image coordinate system can be identified according to the position of the recognition frame.

[0082] It can be understood that the third position is formed based on the positions of different cameras, with the corresponding camera as the origin.

[0083] Since the above-mentioned pedestrian recognition frame is a range rather than a specific position coordinate. Therefore, in some embodiments, the third position can be expressed as the center position of the bottom of the recognition frame. Alternatively, the third position can also be expressed as the center position of the recognition frame. It should be noted that whether the third position adopts the center position of the bottom of the recognition frame or the center position of the recognition frame, the third position selection method of the pedestrian should be consistent in position judgment. For example, the third position of the pedestrian in the first camera adopts the center position of the bottom of the pedestrian recognition frame, and thus the third position of the pedestrian in the other three cameras also adopts the center position of the bottom of the recognition frame.

[0084] Step S520: converting the third position into the first position, where the first position is the position of the pedestrian in the world coordinate system where the engineering machine is located.

[0085] In the above embodiment, the third position may be a position coordinate in an image coordinate system formed by a picture generated by a camera. By installing four cameras around the construction machinery, a 360-degree image monitoring function around the vehicle is realized. Therefore, in this embodiment, the four cameras will transmit the image monitoring screen, and the third position of the pedestrian identified in each screen is based on the camera corresponding to the screen as the center.

[0086] In some embodiments, the third position can be converted into the first position mentioned above by data registration. It can be understood that the first position can be the position of the pedestrian in the world coordinate system where the engineering machine is located. It can be further understood that the world coordinate system where the engineering machine is located is a coordinate system centered on the engineering machine.

[0087] like Figure 6B As shown, the conversion from the third position to the first position may include the following steps:

[0088] Step S630: transform the original image coordinate system into the world coordinate system.

[0089] by Figure 7 Taking the original coordinates shown in the figure as an example, when four cameras are installed around the construction machinery vehicle, the image screen is displayed as follows Figure 7 The left figure shows. Through these four images, the top view of the construction machinery can be obtained through perspective transformation, that is, Figure 7The figure on the right shows.

[0090] The original image coordinates are centered on the camera corresponding to the original image, and the top view coordinates are centered on the construction machinery to establish a world coordinate system. In the original image coordinate system, the unit is set to pixel. In the top view world coordinate system, the unit is set to meter. One pixel in the top view can be set to 1 centimeter in the corresponding world coordinate system.

[0091] The pedestrian identification frame identified in step S610 is as follows: Figure 8 As shown, the bottom center of the recognition frame can be used as the coordinate of the pedestrian in the original image, and its pixel coordinate is A. Through image stitching technology, the pixel coordinate of the pedestrian in the top view can be obtained as B. The position of the pedestrian in the world coordinate system centered on the construction machinery is set to C, in meters.

[0092] In some embodiments, the position C of the pedestrian in the world coordinate system can be realized by C=H*A / 100, where H can be a homogeneous transformation matrix determined by camera calibration. However, since there may be a certain error in camera calibration, and the algorithm selection of the pedestrian recognition frame may also produce some errors, there will be an error between the coordinates of point C and the target position of the pedestrian, generally around 0.2-1m.

[0093] Step S640: Determine the coordinates in the world coordinate system as the first position of the pedestrian.

[0094] In the above embodiment, the first position of the pedestrian may be the position C of the pedestrian in the world coordinate system. It can be understood that the first position of the pedestrian is position information generated by image detection and ultimately based on the construction machinery as the center.

[0095] In some embodiments, the information of the first position of the pedestrian is obtained according to the image detection, and the position information of the pedestrian can be obtained by using an artificial intelligence deep learning method. The artificial intelligence deep learning method, especially the application of convolutional neural network (CNN), can automatically learn and extract key features in the image by training a large amount of labeled data, thereby realizing the recognition and positioning of the target position.

[0096] This application is described by taking a real-time detection transformer (RT-DETR) algorithm as an example.

[0097] In the related art, the RT-DETR algorithm can be used to solve the problem that non-maximum suppression (NMS) post-processing is difficult to optimize and not robust enough. Fig. 9It is a schematic diagram of a model architecture of the RT-DETR algorithm.

[0098] The model architecture can include a backbone network, a hybrid encoder, and a transformer decoder with an auxiliary prediction head.

[0099] The model architecture can use the output features {S3, S4, S5} of the last three stages of the backbone network as the input of the encoder. The hybrid encoder converts multi-scale features into a series of image features through the attention-based intra-scale feature interaction module (AIFI) and the cross-scale feature fusion module (CCFM) based on the neural network.

[0100] In some embodiments, AIFI can be directly composed of convolutional layers, batch normalization layers, and activation layers.

[0101] In some embodiments, CCFM may be composed of a fusion module, and the fusion module may be composed of a stack of convolutional layers, batch normalization layers, and activation layers of different numbers.

[0102] Based on the hybrid encoder composed of the above AIFI and CCFM, the simultaneous interactions within and between scales of each layer can be decoupled, which greatly improves the speed and accuracy of pedestrian recognition.

[0103] In some embodiments, laser radar detection is performed on the working area detected by the above image to determine the second position of the obstacle in the working area. The second position can be obtained by the following steps.

[0104] Fig.10 It is a schematic diagram of a process for obtaining a second position through laser radar detection provided in an embodiment of the present application.

[0105] Step 1: Perform laser radar detection on the working area to determine the fourth position of the obstacle, where the fourth position is the position of the obstacle in the laser radar coordinate system;

[0106] Based on the working area detected by the above image, the laser radar is continued to be used to detect the area. And the position of the obstacle scanned by the laser radar is set as the fourth position. In some embodiments, two laser radar sensors can be installed on the engineering machinery, and the coverage range of each laser radar sensor is 270 degrees, so that a panoramic 360-degree coverage can be achieved. Therefore, when one of the laser radars scans an obstacle, the coordinates of the obstacle centered on the laser radar can be called the fourth position. It can be understood that the fourth position is relative to the position of the laser radar. It will change with the position of the laser radar.

[0107] Step 2: Convert the fourth position into the second position, where the second position is the position of the obstacle in the world coordinate system where the engineering machine is located.

[0108] In the above embodiment, the fourth position may be a position coordinate formed relative to the position of the laser radar. Since in the above embodiment, the engineering machinery uses two laser radars with 270-degree coverage. Therefore, in this embodiment, both laser radars will transmit the results of laser radar detection. The fourth position of the obstacle identified by them is based on the corresponding laser radar as the center.

[0109] like Fig.11A As shown in , the coordinates P1 of the obstacle detected by the laser radar are the coordinates in the laser coordinate system. Fig. 11B As shown, in the world coordinate system where the engineering machine is located, that is, in the coordinate system centered on the engineering machine, the coordinate of the obstacle is P2. In some embodiments, the value of P2 can be obtained by the following formula, that is, P2 = P1 + E. E in the formula can represent the offset of the laser to the world coordinate system.

[0110] It can be understood that the second position determined by the radar laser is the position of the obstacle in the world coordinate system where the engineering machinery is located.

[0111] In some embodiments, the target position of the pedestrian can be tracked by the first position detected by the image and the second position detected by the radar laser. Since the range of radar laser detection is larger than that of image detection, when the position trajectory of the pedestrian exceeds the range of image detection but is still within the range of laser radar detection, the position of the pedestrian can continue to be tracked by the laser radar. That is, it can be understood that when the position of the pedestrian exceeds the range of image detection but is still within the range of laser radar detection, the first position cannot be obtained and only the second position can be obtained. At this time, the target position of the pedestrian is tracked based on the information of the second position.

[0112] Below, Fig.12 The technical solution of the present application is further described with reference to specific embodiments.

[0113] Fig.12 The figure shows a flow chart of tracking the position of pedestrians through image detection and radar laser detection.

[0114] Step S1210: The first position of the pedestrian can be confirmed through image detection, and the second position of the obstacle can be confirmed through laser radar detection.

[0115] Step S1220: When the obstacle is confirmed to be a pedestrian through the first position and the second position, the coordinates of the pedestrian can be determined as the target position. The coordinates of the pedestrian are marked, and when the pedestrian is within the range of image detection and laser radar detection, the position of the pedestrian is tracked based on the marked coordinates.

[0116] Step S1230: Based on the location trajectory of the pedestrian, it can be divided into two situations.

[0117] Case 1: When the pedestrian moves within the image detection range, steps S1210 and S1220 may be repeated, and the coordinate mark of the pedestrian is updated in real time.

[0118] Case 2: When the moving trajectory of the pedestrian exceeds the image detection range but is still within the laser radar detection range, jump to step S1240 and continue to track the pedestrian target through laser.

[0119] Step S1240: Tracking pedestrian targets through laser radar.

[0120] When the pedestrian is beyond the image detection range but still within the laser radar detection range, the laser radar can continue to track the pedestrian.

[0121] Fig.13 A flow chart of position tracking of pedestrians through LiDAR detection is proposed.

[0122] Step S1310: Determine the target position of the pedestrian before it disappears from the detection range.

[0123] Based on the above step S1240, when the position trajectory of the pedestrian has exceeded the image detection range, the position tracking of the pedestrian is started by the laser radar. The coordinates of the pedestrian target when it disappears in the image detection range can be set to X.

[0124] Step S1320: Use the laser radar to detect the position of the obstacle closest to the target position.

[0125] When the position trajectory of the pedestrian exceeds the image detection range, the laser radar can scan the area where X is located based on the position of X, and obtain the coordinate Y of the obstacle scanned in the area.

[0126] Step S1330: If the difference between the obstacle position and the target position satisfies a second preset condition, updating the target position to the obstacle position.

[0127] The difference between the obstacle position and the target position can be understood as the straight-line distance between the first position and the second position. It can be a distance based on a two-dimensional space or a distance based on a three-dimensional space. The difference calculation between the two positions can refer to the difference calculation method between the first position and the second position described above.

[0128] The second preset condition can be understood as a basis for determining whether the obstacle detected by the laser radar is a target pedestrian. The determination of the type of the second preset condition depends on the selection of a method for calculating the difference between the obstacle position and the target position.

[0129] For example, when the second preset condition is set to 0.5, the position difference between the target position coordinate X and the obstacle coordinate Y can be calculated by the norm method, that is, the position difference can be calculated by the following formula, such as norm(X―Y).

[0130] When norm(X―Y)<0.5, it means that the difference between the coordinate X of the target pedestrian and the coordinate Y of the obstacle is less than the second preset condition. This means that at this time, the obstacle detected by the lidar is the target pedestrian. It can also be understood that at this time, the target position of the pedestrian is outside the image detection range, but within the lidar detection range. Therefore, the coordinate value of X in step S1310 can be updated to the coordinate value of Y, and it is considered that the target pedestrian is at the Y position based on the path. Then repeat steps S1310-S1330 to continue pedestrian target tracking.

[0131] When norm(X―Y)>0.5, it means that the difference between the coordinate X of the target pedestrian and the coordinate Y of the obstacle is greater than the second preset condition. This means that at this time, the obstacle detected by the laser radar is not a pedestrian. It can also be understood that at this time, the position trajectory of the pedestrian has also exceeded the detection range of the laser radar. The target tracking process ends.

[0132] By combining image detection and LiDAR detection to track the position of pedestrians, the tracking range of pedestrian position trajectories can be expanded, thereby improving the accuracy of pedestrian positions in the operating area of ​​engineering machinery.

[0133] In some embodiments, the control strategy of the engineering machinery can be determined according to the distance between the target position and the engineering machinery. The above control strategy can realize the precise control of the engineering machinery, and dynamically adjust the operating parameters by real-time monitoring the distance between the target position and the engineering machinery, so as to ensure that the distance between the engineering machinery and pedestrians can be controlled under various operating conditions, thereby improving the safety of the normal operation of the engineering machinery.

[0134] In some embodiments, the moving direction of the pedestrian can be determined based on the distance between the real-time tracking target position and the engineering machinery, thereby determining the control strategy of the engineering machinery, thereby further improving the flexibility of the engineering machinery and the safety of its normal operation.

[0135] The control strategies mentioned in the above embodiments may include one or more of the following: issuing an alarm, limiting the action of the actuator of the engineering machinery, and limiting the moving speed of the engineering machinery.

[0136] The technical solution of the present application is described below in conjunction with specific embodiments.

[0137] By using the first position confirmed by image detection and the second position confirmed by lidar detection, the pedestrian can be identified and the target position of the pedestrian can be determined.

[0138] According to the distance and position trajectory of the pedestrian from the construction machinery, the operation range of the construction machinery can be divided into dangerous area levels, and the control strategy of the construction machinery can be further generated. Table 1 is taken as an example.

[0139] Table 1 Control strategies for pedestrian position trajectories in different areas

[0140]

[0141] Another embodiment of the present application further provides an engineering machine, which includes an image detection system, a laser radar system, and a processing system. The image detection system can be used to perform image detection on the working area of ​​the engineering machine; the laser radar system can be used to perform laser radar detection on the working area; the processing system can be used to execute the identification and positioning method in any of the above embodiments based on the data collected by the image detection system and the laser radar system.

[0142] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0143] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0144] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0145] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0148] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that can be read by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0149] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A recognition and positioning method applied to engineering machinery, characterized in that: include: Performing image detection on the operation area of ​​the construction machinery to determine a first position of a pedestrian in the operation area; Performing laser radar detection on the operation area to determine a second position where an obstacle in the operation area is located; A target position where the pedestrian is located is determined according to the first position and the second position.

2. The method according to claim 1, characterized in that The step of determining a target position of the pedestrian according to the first position and the second position includes: If the difference between the first position and the second position satisfies a first preset condition, determining the second position as the target position; and / or If the difference between the first position and the second position does not satisfy the first preset condition, it is determined that no pedestrian is present in the working area.

3. The method according to claim 1, characterized in that The performing image detection on the working area of ​​the engineering machinery to determine the first position of the pedestrian in the working area includes: Performing image detection on the operation area to determine a third position of the pedestrian, where the third position is the position of the pedestrian in the image coordinate system; The third position is converted into the first position, where the first position is the position of the pedestrian in the world coordinate system where the engineering machine is located.

4. The method according to claim 1, characterized in that: The performing laser radar detection on the working area to determine the second position of the obstacle in the working area includes: Performing laser radar detection on the working area to determine a fourth position of the obstacle, where the fourth position is a position of the obstacle in a laser radar coordinate system; The fourth position is converted into the second position, where the second position is the position of the obstacle in the world coordinate system where the engineering machine is located.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: During the process of tracking the position of the pedestrian, if the pedestrian disappears from the detection range of the image detection, the position of the pedestrian is tracked using a laser radar.

6. The method according to claim 5, characterized in that The method of tracking the position of the pedestrian by using a laser radar comprises: Determine the target position of the pedestrian before it disappears from the detection range; Using the laser radar to detect the position of the obstacle closest to the target position; If the difference between the obstacle position and the target position meets a second preset condition, the target position is updated to the obstacle position.

7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Determining a control strategy for the engineering machinery according to the distance between the target position and the engineering machinery; The engineering machine is controlled according to the control strategy.

8. The method according to claim 7, characterized in that The step of determining the control strategy of the engineering machinery according to the distance between the target position and the engineering machinery comprises: A control strategy for the engineering machinery is determined according to the distance between the target position and the engineering machinery and the moving direction of the pedestrian.

9. The method according to claim 8, characterized in that The control strategy includes one or more of the following: issuing an alarm, limiting the action of the actuator of the engineering machinery, and limiting the moving speed of the engineering machinery.

10. An engineering machine, characterized in that: include: Image detection system, used for image detection of the working area of ​​the construction machinery; A laser radar system for performing laser radar detection on the operation area; A processing system for executing the method as claimed in any one of claims 1 to 9 based on the data collected by the image detection system and the lidar system.