Obstacle recognition method, control device, engineering machinery and machine readable storage medium
By performing depth calculation and edge extraction of pavement images, generating pavement fusion images and determining obstacle attribute information, the problems of inaccurate obstacle identification and computing resource limitation in the prior art are solved, and more efficient and stable obstacle identification is achieved.
Patent Information
- Application Number
- CN202411201793.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2025-01-10
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, deep learning models have inaccurate obstacle attribute information due to lack of diverse training data and changes in external conditions in obstacle recognition scenarios, and the limitation of computing resource makes real-time output difficult, resulting in limited application scenarios.
By performing depth calculation and edge extraction on the pavement image, feature alignment and deep fusion are performed, pavement fusion images are generated, and the location, size and depth information of obstacles are determined, so as to improve the accuracy and efficiency of obstacle attribute information.
It improves the accuracy and acquisition efficiency of barrier attribute information, makes it suitable for a variety of scenarios for rapid identification and accurate identification, reduces dependence on computing resources, and enhances stability under different external conditions.
Smart Images
Figure CN119296070A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image processing in engineering machinery, and in particular to an obstacle recognition method, a control device, an engineering machinery and a machine-readable storage medium. Background Art
[0002] With the rapid development of deep learning technology, deep learning technology has been widely used in image recognition and computer vision. In the recognition of obstacles such as potholes, deep learning technology can identify the characteristics of obstacles such as potholes in road images by training models such as convolutional neural networks. The road image to be detected is input into the trained model. The trained model can process and identify the features in the road image in the hidden layer, thereby distinguishing obstacles from normal roads in the road image, and output obstacle attribute information such as the area and depth of the obstacle. Model training requires a large amount of well-labeled training data. However, due to the lack of sufficient diversity of training data to train the model, the reliability of the trained model is poor.
[0003] In actual obstacle detection scenarios, changes in external conditions such as weather, obstacle type, and road material will affect the output results of the model, and thus it is impossible to obtain accurate obstacle attribute information. When there is a large error in the obstacle attribute information, it is easy to mistake a pothole for a low-depth obstacle, and then it is mistakenly believed that the obstacle does not need to be avoided, resulting in the obstacle attribute information being unable to be applied to scenarios where obstacles need to be accurately identified. In addition, the model calculation process is complex and usually requires a large amount of computing resources. Due to the limitations of the device's processing power, it is difficult for the model to output obstacle attribute information in real time. The efficiency of the device in obtaining obstacle attribute information during high-speed driving is too low, and it is easy to obtain obstacle attribute information only when the device is too close to the obstacle, and thus it is impossible to generate an obstacle avoidance strategy in real time based on the obstacle attribute information. Obstacle attribute information is not applicable to scenarios where obstacles need to be quickly identified and located. The limitations of outputting obstacle attribute information through the model are large, resulting in limited application scenarios for obtaining obstacle attribute information. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide an obstacle identification method, a control device, an engineering machine and a machine-readable storage medium. The obstacle identification method is used to solve the problem of limited application scenarios of obstacle attribute information in the prior art.
[0005] In order to achieve the above-mentioned object, the first aspect of the present application provides an obstacle identification method, which includes:
[0006] Performing depth calculation on the road surface image acquired by the image acquisition device to obtain a road surface depth image;
[0007] Extract the edge of the road surface image to obtain an edge feature image;
[0008] Performing feature alignment and depth fusion on the road surface depth image and the edge feature image, so as to map the depth value in the road surface depth image to the edge feature image, and obtain a road surface fusion image;
[0009] Determine whether there are obstacles on the road surface according to the fused road surface image;
[0010] When there are obstacles on the road surface, an obstacle image is extracted from the road surface fusion image;
[0011] Obtain the attribute information of the obstacle based on the obstacle image.
[0012] In an embodiment of the present application, the attribute information of the obstacle includes location information of the obstacle;
[0013] Obtain the attribute information of the obstacle based on the obstacle image, including:
[0014] Obtain the two-dimensional coordinates of the obstacle image;
[0015] The pixel depth value is determined as the z-axis coordinate value of the obstacle in the three-dimensional coordinate system, wherein the pixel depth value is the depth value corresponding to the pixel in the obstacle image;
[0016] Determine the x-axis coordinate value and the y-axis coordinate value of the obstacle in the three-dimensional coordinate system according to the z-axis coordinate value, the two-dimensional coordinate, the coordinate of the image acquisition device and the focal length of the image acquisition device;
[0017] Based on the x-axis coordinate value, the y-axis coordinate value, and the z-axis coordinate value, the position information of the obstacle is determined.
[0018] In an embodiment of the present application, the attribute information of the obstacle also includes size information of the obstacle;
[0019] Obtain the attribute information of the obstacle based on the obstacle image, including:
[0020] Based on the obstacle image, determining a first width, a first length, and a first height of the obstacle in two-dimensional coordinates;
[0021] According to the focal length of the image acquisition device and the pixel depth value, convert the first width into a second width on the three-dimensional coordinate, convert the first length into a second length on the three-dimensional coordinate, and convert the first height into a second height on the three-dimensional coordinate;
[0022] Based on the second width, the second length, and the second height, size information of the obstacle is determined.
[0023] In an embodiment of the present application, the attribute information of the obstacle also includes depth information of the obstacle;
[0024] Obtain the attribute information of the obstacle based on the obstacle image, including:
[0025] Get the depth values of all pixel blocks in the obstacle image;
[0026] Calculate the average of the depth values corresponding to all pixels in the target pixel block to obtain the depth average, where the target pixel block is the pixel block with the largest depth value;
[0027] The depth information of the obstacle is determined based on the difference between the mean depth value and the edge depth value of the obstacle.
[0028] In an embodiment of the present application, determining whether there is an obstacle on the road surface according to the road surface fusion image includes:
[0029] For any edge contour feature in the road surface fusion image, determine whether the range of the edge contour feature is greater than a preset range according to the length and width of the edge contour feature;
[0030] When the range of the edge contour feature is greater than the preset range, the number of target pixel points in the edge contour feature whose height is greater than the preset height is obtained;
[0031] When the number of target pixel points is greater than a preset number, obtaining a height difference between an inner height and an outer height of the edge contour feature;
[0032] When the height difference is greater than a preset height difference, the edge profile feature is determined as an obstacle in the road surface.
[0033] In an embodiment of the present application, feature alignment and depth fusion are performed on the road surface depth image and the edge feature image to map the depth value in the road surface depth image to the edge feature image to obtain a road surface fusion image, including:
[0034] Obtaining a target edge image in the edge feature image, wherein the edge of the target edge image is a minor arc;
[0035] The road surface depth image and the target edge image are feature aligned and depth fused to map the depth value in the road surface depth image to the target edge image to obtain a road surface fused image.
[0036] In an embodiment of the present application, depth calculation is performed on a road surface image acquired by an image acquisition device to obtain a road surface depth image, including:
[0037] The SGM algorithm is used to calculate the depth of the road surface image collected by the image acquisition device to obtain a road surface depth image.
[0038] A second aspect of the present application provides a control device, comprising:
[0039] a memory configured to store instructions; and
[0040] The processor is configured to call instructions from the memory and implement the above obstacle identification method when executing the instructions.
[0041] A third aspect of the present application provides an engineering machine, comprising:
[0042] An image acquisition device configured to acquire a road surface image;
[0043] and the control device mentioned above.
[0044] A fourth aspect of the present application provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned obstacle identification method.
[0045] The present application provides an obstacle recognition method, which includes: performing depth calculation on a road surface image collected by an image acquisition device to obtain a road surface depth image; performing edge extraction on the road surface image to obtain an edge feature image; performing feature alignment and depth fusion on the road surface depth image and the edge feature image to map the depth value in the road surface depth image to the edge feature image to obtain a road surface fusion image; determining whether there is an obstacle in the road surface according to the road surface fusion image; extracting the obstacle image in the road surface fusion image when there is an obstacle in the road surface; and obtaining the attribute information of the obstacle according to the obstacle image. By analyzing the obstacles in the road surface together with the depth information and edge features in the road surface fusion image, the accuracy of obtaining the attribute information of the obstacle is improved. In addition, compared with the analysis of a large amount of road surface point cloud data, obtaining the attribute information of the obstacle through a two-dimensional obstacle image improves the efficiency of obtaining the attribute information of the obstacle, so that the attribute information of the obstacle is suitable for a variety of scenarios such as requiring rapid and accurate identification of obstacles.
[0046] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:
[0048] Figure 1 A schematic diagram of a process flow of an obstacle identification method according to an embodiment of the present application is schematically shown;
[0049] Figure 2 A schematic structural diagram of an engineering machinery according to an embodiment of the present application is shown schematically. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0051] It should be noted that if the embodiments of the present application involve directional indications (such as up and down), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0052] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0053] Existing technologies usually process collected road images into point cloud data. In actual pothole detection scenarios, the point cloud data is input into the algorithm model. The algorithm model needs to process a large number of redundant point clouds, which usually requires a lot of computing resources. Due to the limitations of the processing power of engineering machinery, it is difficult for the model to output accurate road pothole recognition results in real time.
[0054] See also Figure 1 , Figure 1 The following schematically shows a flow chart of an obstacle identification method according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides an obstacle identification method, which may include the following steps.
[0055] S110, performing depth calculation on the road surface image acquired by the image acquisition device to obtain a road surface depth image.
[0056] The image acquisition device is used to acquire the road surface image, and then the depth of the acquired road surface image is calculated to obtain the road surface depth image. It should be understood that the type of the image acquisition device is set according to actual needs and is not limited here. For ease of understanding, the image acquisition device in the embodiment of the present application is a binocular camera with a high dynamic range (HDR) to obtain the image depth through the image parallax between the left and right eyes of the binocular camera.
[0057] In an embodiment of the present application, depth calculation is performed on a road surface image acquired by an image acquisition device to obtain a road surface depth image, including:
[0058] The SGM algorithm is used to calculate the depth of the road surface image collected by the image acquisition device to obtain a road surface depth image.
[0059] Generally, a block matching algorithm is used to calculate the depth of a road surface image. In this embodiment, the SGM (Semi-Global Matching) algorithm is used to construct cost aggregation and continuity constraints between pixels to calculate the depth of the road surface image collected by the image acquisition device to obtain a road surface depth image. Compared with the block matching algorithm for depth calculation, the SGM algorithm is used for depth calculation in this embodiment, which can better handle image noise and areas with insufficient texture, and the generated depth map is more accurate and stable.
[0060] In this embodiment, a highly parallelized algorithm is used to optimize the SGM algorithm. When the optimized SGM algorithm is used for depth calculation, a road depth image can be obtained efficiently to meet the demand for high frame rate real-time generation of the road depth image.
[0061] S120, extracting edges from the road surface image to obtain an edge feature image.
[0062] When a road surface image is collected, edge extraction is performed on the road surface image. In this embodiment, the road surface images collected by the left and right eyes of the binocular camera are both RGB (Red Green Blue) color images. Edge extraction is performed on the RGB color image of the left eye to obtain edge information of the road surface image and obtain an edge feature image. Using RGB color images for edge detection can obtain more accurate edge features.
[0063] It should be understood that the depth of the road image can be calculated first and then the edge of the road image can be extracted, or the edge of the road image can be extracted first and then the depth of the road image can be calculated, or the edge extraction and depth calculation can be performed on the road image at the same time, which is not limited here.
[0064] S130, performing feature alignment and depth fusion on the road surface depth image and the edge feature image, so as to map the depth value in the road surface depth image to the edge feature image, and obtain a road surface fusion image.
[0065] The road depth image and the edge feature image are feature aligned to map the information of the two data sources in the road depth image and the edge feature image to the same feature space. At the same time, the road depth image and the edge feature image are feature aligned to map the depth value in the road depth image to the edge feature image to obtain a road fusion image.
[0066] In an embodiment of the present application, feature alignment and depth fusion are performed on the road surface depth image and the edge feature image to map the depth value in the road surface depth image to the edge feature image to obtain a road surface fusion image, including:
[0067] Obtaining a target edge image in the edge feature image, wherein the edge of the target edge image is a minor arc;
[0068] The road surface depth image and the target edge image are feature aligned and depth fused to map the depth value in the road surface depth image to the target edge image to obtain a road surface fused image.
[0069] The edge feature image includes all edge information in the road surface image. Usually, the edge of obstacles such as potholes is a semi-envelope or fully-envelope minor arc. All target edge images in the edge feature image are obtained, where the edge of the target edge image is a minor arc. The road depth image and the target edge image are feature aligned and deeply fused to map the depth value in the road depth image to the target edge image to obtain a road fusion image. For any edge feature in the road fusion image, if the depth value of the edge feature changes significantly, it is determined that the edge feature is likely to be the edge corresponding to the obstacle.
[0070] Compared with processing the complete road surface image, when processing the road surface fusion image, it is only necessary to determine whether the target edge image is the image corresponding to the obstacle, which reduces the amount of data to be processed and further improves the efficiency of determining the attribute information of the obstacle.
[0071] S140, determining whether there is an obstacle on the road surface based on the fused road surface image.
[0072] The depth of the edge features in the road surface can be quickly determined by the road surface fusion image. Based on the road surface fusion image, a preliminary check is performed on the obstacles in the road surface to determine whether there are obstacles in the road surface, so as to generate a road surface recognition result. The road surface recognition result includes the presence of obstacles in the road surface and the absence of obstacles in the road surface.
[0073] When the road surface recognition result is not real-time, the obstacle avoidance strategy generated based on the road surface recognition result is also not real-time, which makes the obstacle avoidance strategy unsuitable for actual scenarios such as autonomous walking of construction machinery. Compared with processing point cloud data, in this embodiment, road obstacles are detected directly through road surface fusion images in the form of two-dimensional images, which reduces the amount of redundant data processing and can quickly obtain road surface recognition results with high real-time performance.
[0074] In an embodiment of the present application, determining whether there is an obstacle on the road surface according to the road surface fusion image includes:
[0075] For any edge contour feature in the road surface fusion image, determine whether the range of the edge contour feature is greater than a preset range according to the length and width of the edge contour feature;
[0076] When the range of the edge contour feature is greater than the preset range, the number of target pixel points in the edge contour feature whose height is greater than the preset height is obtained;
[0077] When the number of target pixel points is greater than a preset number, obtaining a height difference between an inner height and an outer height of the edge contour feature;
[0078] When the height difference is greater than a preset height difference, the edge profile feature is determined as an obstacle in the road surface.
[0079] The width, length and height of each edge contour in the road fusion image in the two-dimensional plane are obtained respectively. For any edge contour feature in the road fusion image, the area of the edge contour feature is calculated according to the length and width of the edge contour feature in the two-dimensional plane. According to the area of the edge contour feature, it is determined whether the range of the edge contour feature is greater than the preset range.
[0080] When the range of the edge contour feature is greater than the preset range, the number of target pixel points in the edge contour feature whose height is greater than the preset height is obtained. Specifically, since a single pixel cannot fully represent the depth of the obstacle, a 3*3 mean filter convolution kernel is used in this embodiment to perform a convolution operation on the edge contour feature, and the number of target pixel points in the edge contour feature whose height is greater than the preset height is counted. When the number of target pixel points is greater than the preset number, the height difference between the inner height and the outer height of the edge contour feature is obtained. When the height difference is greater than the preset height difference, the edge contour feature is determined to be an obstacle in the road surface.
[0081] It should be understood that the preset range, preset number and preset height difference are all set according to actual needs and are not limited here. When the range of the edge contour feature is greater than the preset range, the number of target pixels is greater than the preset number, and the height difference is greater than the preset height difference, the edge contour feature is determined as an obstacle in the road surface.
[0082] S150: When there is an obstacle on the road surface, extract the obstacle image from the road surface fusion image.
[0083] When the road surface recognition result indicates that there is an obstacle in the road surface, the obstacle image in the road surface fusion image is extracted to process the obstacle image more accurately and obtain accurate attribute information of the obstacle. It should be understood that the attribute information of the obstacle is set according to actual needs, and can be the area and depth of the obstacle, etc., which is not limited here.
[0084] It should be understood that obstacles on the road can be upwardly protruding obstacles or downwardly sunken potholes, which will not be elaborated here.
[0085] S160: Obtain attribute information of the obstacle according to the obstacle image.
[0086] When the road surface image captured by the image acquisition device is a two-dimensional image, the extracted obstacle image is also a two-dimensional image. The three-dimensional obstacle is measured and located according to the obstacle image, thereby obtaining the attribute information of the obstacle in the three-dimensional space.
[0087] By analyzing obstacles on the road surface by combining the depth information and edge features in the road surface fusion image, a more accurate obstacle image can be obtained, which improves the accuracy of the obstacle attribute information obtained, and the obstacle attribute information obtained is applicable to a variety of scenarios. In addition, compared with the analysis of a large amount of road surface point cloud data, obtaining the obstacle attribute information through a two-dimensional obstacle image improves the efficiency of obtaining the obstacle attribute information, and further makes the obstacle attribute information applicable to a variety of scenarios that require rapid and accurate obstacle identification.
[0088] In an embodiment of the present application, the attribute information of the obstacle includes location information of the obstacle;
[0089] Obtain the attribute information of the obstacle based on the obstacle image, including:
[0090] Obtain the two-dimensional coordinates of the obstacle image;
[0091] The pixel depth value is determined as the z-axis coordinate value of the obstacle in the three-dimensional coordinate system, wherein the pixel depth value is the depth value corresponding to the pixel in the obstacle image;
[0092] Determine the x-axis coordinate value and the y-axis coordinate value of the obstacle in the three-dimensional coordinate system according to the z-axis coordinate value, the two-dimensional coordinate, the coordinate of the image acquisition device and the focal length of the image acquisition device;
[0093] Based on the x-axis coordinate value, the y-axis coordinate value, and the z-axis coordinate value, the position information of the obstacle is determined.
[0094] When the road image captured by the image acquisition device is a two-dimensional image, the extracted obstacle image is also a two-dimensional image, and the length and width of the obstacle obtained in the obstacle image are only the pixel positions in the image. In the scenario of autonomous driving of construction machinery, the obstacle avoidance and navigation of the construction machinery require the three-dimensional position of the obstacle in an absolute scale, and the three-dimensional position of the obstacle is located and measured based on the center point of the obstacle.
[0095] For ease of understanding, the two-dimensional coordinates of the obstacle image obtained in the implementation of this application are (u, v), and the two-dimensional coordinates need to be converted into three-dimensional coordinates in the three-dimensional coordinate system of the image acquisition device. Specifically, the pixel depth value is the depth value corresponding to the pixel in the obstacle image, and each pixel in the obstacle image corresponds to a depth value. The pixel depth value is determined as the z-axis coordinate value of the obstacle in the three-dimensional coordinate system. According to the z-axis coordinate value, the two-dimensional coordinate, the coordinates of the image acquisition device and the focal length of the image acquisition device, the x-axis coordinate value and the y-axis coordinate value of the obstacle in the three-dimensional coordinate system are determined respectively:
[0096]
[0097]
[0098] Among them, X is the x-axis coordinate, Y is the y-axis coordinate, u and v are the two-dimensional coordinates of the obstacle image, C x and C y are the center point coordinates of the image acquisition device, f x and f y are the focal lengths of the image acquisition device, and Z is the depth value of the pixel point.
[0099] Based on the x-axis coordinate value, the y-axis coordinate value and the z-axis coordinate value, the position information of the obstacle is determined, that is, the three-dimensional position of the obstacle in the coordinate system of the image acquisition device is determined to be (X, Y, Z).
[0100] In an embodiment of the present application, the attribute information of the obstacle also includes size information of the obstacle;
[0101] Obtain the attribute information of the obstacle based on the obstacle image, including:
[0102] Based on the obstacle image, determining a first width, a first length, and a first height of the obstacle in two-dimensional coordinates;
[0103] According to the focal length of the image acquisition device and the pixel depth value, convert the first width into a second width on the three-dimensional coordinate, convert the first length into a second length on the three-dimensional coordinate, and convert the first height into a second height on the three-dimensional coordinate;
[0104] Based on the second width, the second length, and the second height, size information of the obstacle is determined.
[0105] Based on the obstacle image, determine the first width, first length and first height of the obstacle in two-dimensional coordinates, that is, determine the width, length and height of the obstacle in the image plane. According to the focal length and pixel depth value of the binocular image acquisition device, convert the first width into the second width in three-dimensional coordinates, convert the first length into the second length in three-dimensional coordinates, and convert the first height into the second height in three-dimensional coordinates. For ease of understanding, the second width is obtained as an example in the embodiments of the present application:
[0106] width=W pixet ×Z / f x (3)
[0107] Among them, width is the second width, W pixel is the first width, f x is the focal length of the image acquisition device, and Z is the depth value of the pixel point.
[0108] It should be understood that the size of obstacles on the actual road surface is irregular. For ease of understanding, in the embodiment of the present application, based on the second width, second length and second height in the image acquisition device coordinate system, a rectangular parallelepiped is constructed as the size information of the obstacle in the image acquisition device coordinate system. In this embodiment, the size information includes the area and depth of the obstacle. According to the area and depth of the obstacle, it is determined whether the engineering machinery needs to avoid the obstacle during driving, and then the autonomous driving route of the engineering machinery is generated.
[0109] When the obstacle is small in area or low in depth, it can be determined that it is not necessary to avoid the obstacle based on the obstacle size information. When the obstacle is large in area and / or high in depth, it can be determined that it is necessary to avoid the obstacle based on the obstacle size information. Highly accurate obstacle attribute information avoids the situation where it is mistakenly believed that the obstacle does not need to be avoided. Obtaining obstacle attribute information can be applied to scenarios that require accurate identification of obstacles, such as obstacle avoidance in autonomous driving.
[0110] In an embodiment of the present application, the attribute information of the obstacle also includes depth information of the obstacle;
[0111] Obtain the attribute information of the obstacle based on the obstacle image, including:
[0112] Get the depth values of all pixel blocks in the obstacle image;
[0113] Calculate the average of the depth values corresponding to all pixels in the target pixel block to obtain the depth average, where the target pixel block is the pixel block with the largest depth value;
[0114] The depth information of the obstacle is determined based on the difference between the depth mean and the edge depth value of the obstacle. Usually, the irregularity of the obstacle and the local depth error of the depth image easily lead to a large error in the depth value of a single pixel representing the depth of the obstacle. In this embodiment, the depth of the obstacle is estimated by the depth of the pixel block. Specifically, for any obstacle image, all pixel blocks corresponding to the obstacle image are obtained, and the depth values of all pixel blocks are obtained. The pixel block with the largest depth value among all pixel blocks is determined as the target pixel block, and the depth values corresponding to all pixel points in the target pixel block are averaged to obtain the depth mean. Based on the difference between the depth mean and the edge depth value of the obstacle, the depth information of the obstacle is determined according to the trigonometric function relationship.
[0115] It should be understood that both the depth information and the size information of the obstacle can be used to determine the depth value of the obstacle. The depth information of the obstacle is used to determine the depth of the obstacle in the world coordinate system, and the size information of the obstacle is used to determine the depth of the obstacle in the image acquisition device coordinate system.
[0116] The present application provides an obstacle recognition method, which includes: performing depth calculation on a road surface image collected by an image acquisition device to obtain a road surface depth image; performing edge extraction on the road surface image to obtain an edge feature image; performing feature alignment and depth fusion on the road surface depth image and the edge feature image to map the depth value in the road surface depth image to the edge feature image to obtain a road surface fusion image; determining whether there is an obstacle in the road surface according to the road surface fusion image; extracting the obstacle image in the road surface fusion image when there is an obstacle in the road surface; and obtaining the attribute information of the obstacle according to the obstacle image. By analyzing the obstacles in the road surface together with the depth information and edge features in the road surface fusion image, the accuracy of obtaining the attribute information of the obstacle is improved. In addition, compared with the analysis of a large amount of road surface point cloud data, obtaining the attribute information of the obstacle through a two-dimensional obstacle image improves the efficiency of obtaining the attribute information of the obstacle, so that the attribute information of the obstacle is suitable for a variety of scenarios such as requiring rapid and accurate identification of obstacles.
[0117] The present application also provides a control device, including:
[0118] a memory configured to store instructions; and
[0119] The processor is configured to call instructions from the memory and implement the above obstacle identification method when executing the instructions.
[0120] See also Figure 2 , Figure 2 The structure diagram of an engineering machine according to an embodiment of the present application is schematically shown. Figure 2 As shown, the embodiment of the present application further provides an engineering machine 200, including:
[0121] An image acquisition device 210 is configured to acquire a road surface image;
[0122] and the control device 220 mentioned above.
[0123] When using models such as convolutional neural networks to identify obstacles on the road and output the attribute information of the obstacles, it is usually necessary to rely on a large amount of training data, otherwise the attribute information of the output obstacles will be inaccurate. In addition, in the main driving scene of the construction machinery 200, the point cloud data of the road surface is usually input to the model. Using the algorithm model to output the attribute information of the obstacle requires a large amount of GPU (Graphics Processing Unit) resources and a large amount of redundant point cloud data needs to be processed. Since the control device 220 of the construction machinery 200 is usually a device with limited computing power such as a mobile device or a low-cost embedded system, the efficiency of the attribute information obtained by the control device 220 is too low, resulting in limited application scenarios of the obstacle attribute information.
[0124] The image acquisition device 210 in this embodiment is configured to acquire a road surface image. The control device 220 is configured to obtain an obstacle image based on the road surface image, and then obtain the attribute information of the obstacle based on the obstacle image. Compared with the analysis of a large amount of road surface point cloud data, the control device 220 obtains the attribute information of the obstacle directly based on the two-dimensional image, which improves the efficiency of obtaining the attribute information of the obstacle, and further makes the attribute information of the obstacle suitable for a variety of scenarios such as the need to quickly identify obstacles and accurately identify obstacles. The control device 220 calculates the parallax between pixels by minimizing the overall energy, thereby generating a road surface depth image. When using deep learning models such as CNN (Convolutional Neural Networks) to identify road obstacles, it is usually necessary to rely on GPU (Graphics Processing Unit) for model training and reasoning. The control device 220 in this application does not require GPU acceleration, and can directly perform deep calculations on the CPU, so that the obstacle recognition method in this embodiment can be applied to scenarios where the computing power of the device is limited.
[0125] The number of annotated images will affect the performance of the deep learning model. A large number of annotated images are required as training data during the training process of the deep learning model, which requires a large amount of resources and time to obtain a sufficient number of annotated images. At the same time, the quality of the annotated images will also affect the performance of the deep learning model. Inaccurate annotations in the annotated images will greatly reduce the performance of the deep learning model, and thus make it impossible for the deep learning model to output accurate attribute information of the obstacle. The control device 220 in this embodiment can adapt to different image acquisition devices 210 and different on-site environments, and is not limited by the quantity and quality of training data when obtaining the attribute information of the obstacle, so that the attribute information of the obstacle is suitable for a variety of scenarios such as the need to quickly identify obstacles and accurately identify obstacles.
[0126] In addition, the performance of the deep learning model is also affected by changes in training data. When external conditions such as weather, road materials, and obstacle types change, the road image input to the model differs greatly from the training data, which will cause the model's performance to drop sharply and fail to output accurate obstacle attribute information. The control device 220 directly identifies and locates obstacles on the road through road images. Road obstacle identification has higher stability and robustness than model identification. The performance of the control device 220 is not affected by training data, avoiding the impact of changes in training data, and can be applied to road obstacle identification under different external conditions. The control device 220 has better generalization capabilities, and can still efficiently obtain accurate obstacle attribute information for unprocessed road conditions.
[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0131] In a typical configuration, the control device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0132] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a control device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0134] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0135] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. An obstacle identification method, characterized in that: The obstacle identification method comprises: Performing depth calculation on the road surface image acquired by the image acquisition device to obtain a road surface depth image; Extracting edges of the road surface image to obtain an edge feature image; Performing feature alignment and depth fusion on the road surface depth image and the edge feature image, so as to map the depth value in the road surface depth image to the edge feature image, and obtain a road surface fusion image; Determining whether there is an obstacle on the road surface according to the road surface fusion image; In the case where there is an obstacle in the road surface, extracting an obstacle image from the road surface fused image; According to the obstacle image, attribute information of the obstacle is acquired.
2. The method according to claim 1, characterized in that The attribute information of the obstacle includes location information of the obstacle; The acquiring attribute information of the obstacle according to the obstacle image includes: Acquiring the two-dimensional coordinates of the obstacle image; Determine the pixel depth value as the z-axis coordinate value of the obstacle in the three-dimensional coordinate system, wherein the pixel depth value is the depth value corresponding to the pixel in the obstacle image; Determine the x-axis coordinate value and the y-axis coordinate value of the obstacle in the three-dimensional coordinate system according to the z-axis coordinate value, the two-dimensional coordinate, the coordinate of the image acquisition device and the focal length of the image acquisition device; Based on the x-axis coordinate value, the y-axis coordinate value, and the z-axis coordinate value, the position information of the obstacle is determined.
3. The method according to claim 2, characterized in that The attribute information of the obstacle also includes size information of the obstacle; The acquiring attribute information of the obstacle according to the obstacle image includes: Determine, based on the obstacle image, a first width, a first length, and a first height of the obstacle in two-dimensional coordinates; According to the focal length of the image acquisition device and the pixel depth value, converting the first width into a second width on the three-dimensional coordinate, converting the first length into a second length on the three-dimensional coordinate, and converting the first height into a second height on the three-dimensional coordinate; Based on the second width, the second length, and the second height, size information of the obstacle is determined.
4. The method according to claim 1, characterized in that: The attribute information of the obstacle includes depth information of the obstacle; The acquiring attribute information of the obstacle according to the obstacle image includes: Obtaining depth values of all pixel blocks in the obstacle image; Calculate the average of the depth values corresponding to all pixels in the target pixel block to obtain a depth average, wherein the target pixel block is the pixel block with the largest depth value; Depth information of the obstacle is determined based on a difference between the depth mean value and an edge depth value of the obstacle.
5. The method according to claim 1, characterized in that The determining whether there is an obstacle on the road surface according to the road surface fusion image includes: For any edge contour feature in the road surface fusion image, determine whether a range of the edge contour feature is greater than a preset range according to the length and width of the edge contour feature; When the range of the edge contour feature is greater than the preset range, obtaining the number of target pixel points in the edge contour feature whose height is greater than the preset height; When the number of the target pixel points is greater than a preset number, obtaining a height difference between an inner height and an outer height of the edge contour feature; When the height difference is greater than a preset height difference, the edge profile feature is determined as an obstacle in the road surface.
6. The method according to claim 1, characterized in that The step of performing feature alignment and depth fusion on the road surface depth image and the edge feature image to map the depth value in the road surface depth image to the edge feature image to obtain a road surface fusion image includes: Acquire a target edge image in the edge feature image, wherein the edge of the target edge image is a minor arc; The road surface depth image and the target edge image are feature aligned and depth fused to map the depth value in the road surface depth image to the target edge image to obtain a road surface fused image.
7. The method according to claim 1, characterized in that The step of performing depth calculation on the road surface image acquired by the image acquisition device to obtain the road surface depth image comprises: The SGM algorithm is used to calculate the depth of the road surface image collected by the image acquisition device to obtain a road surface depth image.
8. A control device, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the obstacle identification method according to any one of claims 1 to 7 when executing the instructions.
9. An engineering machine, characterized in that: include: An image acquisition device configured to acquire a road surface image; And a control device according to claim 8.
10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for causing a machine to execute the obstacle identification method according to any one of 1 to 7.