Bulldozer intelligent obstacle avoidance system based on machine vision and operation method

Abstract

The invention relates to the technical field of visual intelligent obstacle avoidance, in particular to a bulldozer intelligent obstacle avoidance system based on machine vision and an operation method. The method comprises the following steps: at each acquisition moment in the operation process of the bulldozer, acquiring an initial obstacle avoidance coefficient of each obstacle type object according to the occurrence condition and the obstacle avoidance marking condition of each obstacle type object in a similar historical working scene; and then analyzing and acquiring the dynamic obstruction index of each object at each acquisition moment, and then acquiring the obstacle avoidance coefficient of each object in the corresponding environment image at each acquisition moment, thereby determining the target type of the object and planning the obstacle avoidance operation. According to the method, the historical obstacle avoidance condition of the bulldozer in the similar working scene and the dynamic information of each object are combined to comprehensively evaluate the obstacle avoidance coefficient of each object, so that the obstacle avoidance target and the operation target are accurately distinguished to plan the obstacle avoidance operation, and the intelligent obstacle avoidance effect and the operation efficiency of the bulldozer are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual intelligent obstacle avoidance, and in particular to a bulldozer intelligent obstacle avoidance system and operating method based on machine vision. Background Art

[0002] During the operation of the bulldozer, intelligent obstacle avoidance based on machine vision can monitor and analyze the surrounding environment in real time, automatically identify obstacles and work targets, and help the bulldozer automatically avoid obstacles and plan the operation path, avoiding accidental collisions, equipment damage and other accidents caused by operator negligence or visual blind spots. It can not only improve the safety of the bulldozer during driving, but also effectively improve work efficiency.

[0003] However, the bulldozer's operating environment is complex, especially when performing operations such as land leveling and demolition. The bulldozer often needs to actively move forward and clear obstacles of the operating target type. However, during the clearing process, the operating target type obstacles may undergo dynamic changes, such as changes in position or state (such as soil blocks breaking and rolling down). In this case, the existing intelligent obstacle avoidance system may mistakenly identify them as obstacle avoidance targets (such as pedestrians and vehicles) and avoid the operation, resulting in poor intelligent obstacle avoidance effect of the bulldozer, which in turn affects the bulldozer's operating efficiency. Summary of the Invention

[0004] In order to solve the technical problem of poor intelligent obstacle avoidance effect of bulldozers, which in turn leads to low operating efficiency, the purpose of the present invention is to provide a bulldozer intelligent obstacle avoidance system and operating method based on machine vision. The technical solutions adopted are as follows:

[0005] An intelligent obstacle avoidance system for a bulldozer based on machine vision, the system comprising:

[0006] Scene acquisition module: used to obtain environmental images at each acquisition moment during bulldozer operation and mark the obstacle type of each object in the image; determine the working scene of the bulldozer at each acquisition moment and obtain a historical operation database for each working scene. The historical operation database includes at least the obstacle avoidance mark of each object in each historical environmental image;

[0007] Obstacle analysis module: used to obtain the initial obstacle avoidance coefficient of each obstacle type object in the corresponding work scene at each acquisition moment based on the appearance and obstacle avoidance marking of each obstacle type object in the historical operation database; obtain the dynamic obstruction index of each object at each acquisition moment based on the changes in the environmental image at adjacent acquisition moments; at each acquisition moment, obtain the obstacle avoidance coefficient of each object in the corresponding environmental image based on the dynamic obstruction index of each object and the initial obstacle avoidance coefficient of the obstacle type to which each object belongs;

[0008] Obstacle avoidance planning module: used to determine the target type of the object according to the obstacle avoidance coefficient of each object at each collection moment, and plan the obstacle avoidance operation in real time.

[0009] Furthermore, the method for determining the working scenario includes:

[0010] In each environment image, the environment complexity coefficient is obtained based on the shape, size, physical hardness, and proportion of each obstacle type in the environment image;

[0011] According to the environmental complexity coefficient of each environmental image, the working scene of the bulldozer at the corresponding acquisition moment is determined.

[0012] Furthermore, the method for obtaining the environmental complexity coefficient includes:

[0013] For each obstacle type, the area ratio of all objects in the environment image is used as the obstacle density, the average area of ​​all objects is used as the shape size, and the product of the shape size and the preset physical hardness is used as the obstacle parameter;

[0014] The obstacle density and the obstruction parameter of each obstacle type object are combined to obtain an environmental complexity coefficient.

[0015] Furthermore, the method for obtaining the working scene of the bulldozer at the corresponding collection time according to the environmental complexity coefficient includes:

[0016] When the environment complexity coefficient is less than or equal to a first preset threshold, it is determined to be in a low-complexity working scene;

[0017] When the environment complexity coefficient is greater than a first preset threshold and less than a second preset threshold, it is determined to be a medium-complexity working scene;

[0018] When the environment complexity coefficient is greater than or equal to a second preset threshold, it is determined to be in a high-complexity work scene;

[0019] The first preset threshold is smaller than the second preset threshold.

[0020] Furthermore, the method for obtaining the initial obstacle avoidance coefficient includes:

[0021] In the historical operation database corresponding to the work scenario at each collection moment, any obstacle type is used as the target type;

[0022] The ratio of the number of obstacle avoidance marks for the target type object to the total number of obstacle avoidance marks for all objects is used as the obstacle avoidance ratio of the target type object; the ratio of the number of obstacle avoidance marks for the target type object to the number of occurrences of the target type object is used as the obstacle avoidance frequency of the target type object;

[0023] The obstacle avoidance ratio and the obstacle avoidance frequency are integrated to obtain historical obstacle avoidance parameters of the target type object; a negative correlation mapping result of the occurrence frequency of the target type object is used as a confidence weight, the historical obstacle avoidance parameters are weighted using the confidence weight, and the weighted result is used as the initial obstacle avoidance parameter of the target type object.

[0024] Furthermore, the method for obtaining the dynamic obstacle index includes:

[0025] Perform frame difference between the environment image at each acquisition moment and the previous adjacent acquisition moment, and obtain the dynamic change index at each acquisition moment based on the total number of pixels that have changed between the same positions;

[0026] Obtain a point cloud image at each acquisition moment and the position coordinates of each object in the corresponding point cloud image; match the objects in the point cloud image corresponding to each acquisition moment with those in the previous adjacent acquisition moment, and obtain the movement direction of the corresponding object based on the difference in the position coordinates of the matched objects;

[0027] According to the difference between the moving direction of each object and the moving direction of the bulldozer, the dynamic threat parameter of each object is obtained; and the dynamic threat parameter is integrated with the dynamic change index to obtain the dynamic obstruction index of each object at each collection moment.

[0028] Furthermore, the method for obtaining the obstacle avoidance coefficient includes:

[0029] At each acquisition moment, the dynamic obstruction index of each object is normalized, and the normalized value is used as the obstacle weight. The initial obstacle avoidance coefficient of the object of the obstacle type to which the corresponding object belongs is weighted using the obstacle weight, and the normalized result of the weighted result is used as the obstacle avoidance coefficient of the corresponding object.

[0030] Furthermore, the method of determining whether an object is an obstacle target based on the obstacle avoidance coefficient of each object and planning an obstacle avoidance path includes:

[0031] In the environmental image at each acquisition moment, objects with the obstacle avoidance coefficient greater than a preset coefficient threshold are regarded as obstacle targets, and the remaining objects are regarded as operation targets;

[0032] Path planning for obstacle avoidance operations is performed based on all obstacle targets and all operation targets in the environment image.

[0033] Furthermore, the preset coefficient threshold is 0.5.

[0034] A bulldozer intelligent operation method based on machine vision is disclosed. The method utilizes a bulldozer intelligent obstacle avoidance system based on machine vision to plan the bulldozer's obstacle avoidance operation in real time.

[0035] The present invention has the following beneficial effects:

[0036] The present invention first obtains an environmental image at each acquisition moment during bulldozer operation and marks the obstacle type of each object therein; determines the working scene of the bulldozer at each acquisition moment, and obtains a historical operation database for each working scene to provide a historical reference for obstacle avoidance in subsequent working scenes; obtains an initial obstacle avoidance coefficient for each obstacle type object in the corresponding working scene at each acquisition moment based on the appearance and obstacle avoidance marking of each obstacle type object in the historical operation database, and the initial obstacle avoidance coefficient is based on the historical obstacle avoidance reference in similar working scenes and preliminarily reflects the possibility of the bulldozer avoiding the obstacle type object; then, based on the changes in the environmental image at adjacent acquisition moments, obtains a dynamic obstruction index for each object at each acquisition moment, and the dynamic obstruction index reflects the possibility of an obstacle caused by dynamic changes of the object; at each acquisition moment, obtains an obstacle avoidance coefficient for each object in the corresponding environmental image based on the dynamic obstruction index of each object and the initial obstacle avoidance coefficient of the obstacle type object to which each object belongs; at each acquisition moment, determines the target type of the object based on the obstacle avoidance coefficient of each object, and plans the obstacle avoidance operation in real time. The present invention first determines the working scene of the bulldozer, and then evaluates the obstacle avoidance possibility of objects of each obstacle type based on historical operating conditions in similar working scenes. Then, the obstacle avoidance coefficient of each object is comprehensively evaluated based on the dynamic obstruction conditions of each object, thereby accurately distinguishing obstacle avoidance targets and operating targets to plan obstacle avoidance operations, thereby improving the intelligent obstacle avoidance effect and operating efficiency of the bulldozer. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A system module diagram of a machine vision-based intelligent obstacle avoidance system for a bulldozer provided by one embodiment of the present invention;

[0039] Figure 2 A flow chart of a method for obtaining a dynamic obstruction index provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0040] To further illustrate the technical means and effectiveness of the present invention in achieving its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation, structure, features, and effectiveness of a machine vision-based intelligent obstacle avoidance system and operating method for bulldozers proposed in accordance with the present invention. In the following description, references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0041] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0042] The specific scheme of the bulldozer intelligent obstacle avoidance system and operation method based on machine vision provided by the present invention is described in detail below with reference to the accompanying drawings.

[0043] See also Figure 1 , which shows a system module diagram of a bulldozer intelligent obstacle avoidance system based on machine vision provided by an embodiment of the present invention, including a scene acquisition module 101, an obstacle analysis module 102 and an obstacle avoidance planning module 103.

[0044] Scene acquisition module 101: used to obtain the environmental image at each acquisition moment during the bulldozer operation process and mark the obstacle type of each object in it; determine the working scene of the bulldozer at each acquisition moment, and obtain a historical operation database for each working scene. The historical operation database includes at least the bulldozer's obstacle avoidance mark for each object in each historical environmental image.

[0045] In one embodiment of the present invention, a high-definition visual sensor and a laser radar are installed on top of the bulldozer's cab to detect the surrounding working environment while the bulldozer is moving or operating. The visual sensor is used to capture a wide-angle image of the environment in front of the bulldozer, i.e., the environment image. The laser radar is used to scan the bulldozer's working environment 360 degrees, generating a point cloud image. This provides geometric and orientation information of each object in the environment image in three-dimensional space, facilitating the distinction between working targets and obstacles. The acquisition frequency of the visual sensor and the laser radar is set to 10 times per second, but the implementer can also customize the acquisition frequency.

[0046] At each acquisition moment, an environmental image is captured, and then each environmental image is segmented based on semantic segmentation to determine the obstacle type of each object in the environmental image. Furthermore, the obstacle type of an object varies depending on the bulldozer operation scenario: for example, in a mining scenario, objects that appear are mostly ores, clods of soil, and rocks; in a construction site scenario, objects that appear are mostly workers and building materials such as pipelines, steel bars, and cement; and in a land mining scenario, objects that appear are mostly weeds, clods of soil, and trees. Implementers can treat each object as an obstacle type, or they can define their own obstacle types and classify them themselves.

[0047] It should be noted that implementers need to label objects and their obstacle types in a large number of environmental images of different work scenarios in order to train the model for semantic segmentation; the determination and labeling of obstacle types, the training and application of semantic segmentation models are all existing technologies well known to those skilled in the art and will not be repeated here.

[0048] After acquiring the environmental image, the working scene of the bulldozer at the time of acquisition can be determined. In one embodiment of the present invention, the working scenes of the bulldozer are mainly divided into three categories, namely high-complexity working scenes, medium-complexity working scenes, and low-complexity working scenes. In the same type of working scenes, the types of obstacles are similar, and the complexity of the bulldozer's obstacle avoidance operation is also similar.

[0049] Among them, high-complexity work scenarios mainly include mining, post-disaster recovery and cleanup, etc., in which the obstacles are relatively large in size and high in hardness; medium-complexity work scenarios mainly include construction sites, deforestation and land development, etc., in which the obstacles are relatively large in size and moderate in hardness; low-complexity work scenarios include road construction and maintenance, agricultural work, etc., in which the obstacles are small in size and relatively low in hardness.

[0050] Preferably, in one embodiment of the present invention, the method for determining the working scene includes:

[0051] In each environmental image, the environmental complexity coefficient is obtained based on the shape, size, physical hardness, and proportion of each obstacle type in the environmental image; based on the environmental complexity coefficient of each environmental image, the working scene of the bulldozer at the corresponding acquisition time is determined.

[0052] In a preferred embodiment of the present invention, the method for obtaining the environmental complexity coefficient includes:

[0053] For each obstacle type, the area ratio of all objects in the environment image is used as the obstacle density, the average area of ​​all objects is used as the shape size, and the product of the shape size and the preset physical hardness is used as the obstruction parameter. The obstacle density and obstruction parameters of objects of each obstacle type are combined to obtain the environmental complexity coefficient.

[0054] As an example, the total number of pixels is used to measure the area, so as to obtain the obstacle density of each obstacle type in the environmental image; then the shape and size of each obstacle type are obtained, and the shape and size reflect the volume and degree of obstacle of the object; further, the product of the shape and size and the preset physical hardness of the obstacle type object is used as the obstacle parameter; finally, under each obstacle type, the product of the obstacle density and the obstacle parameter is used as the complex sub-parameter, and then the mean of the complex sub-parameters under all obstacle types is mapped to the sigmoid function for normalization, and the normalized value is used as the environmental complexity parameter represented by the corresponding environmental image to facilitate subsequent judgment of the working scenario.

[0055] It should be noted that the preset physical hardness is determined by the physical hardness parameters of each obstacle type object, which is the average hardness of the obstacle type object. It can be obtained by referring to existing information. It is already existing technology and will not be repeated here. Implementers can also use other normalization methods.

[0056] In a preferred embodiment of the present invention, a method for obtaining the working scene of the bulldozer at the corresponding acquisition time according to the environmental complexity coefficient includes:

[0057] When the environmental complexity coefficient is less than or equal to a first preset threshold, it is determined that the bulldozer is in a low-complexity working scene;

[0058] When the environmental complexity coefficient is greater than a first preset threshold and less than a second preset threshold, it is determined that the bulldozer is in a medium-complexity working scene;

[0059] When the environmental complexity coefficient is greater than or equal to the second preset threshold, it is determined that the bulldozer is in a high-complexity working scene.

[0060] As an example, the first preset threshold is 0.3 and the second preset threshold is 0.6. The implementer may also customize it, but it should be noted that the first preset threshold is smaller than the second preset threshold.

[0061] In other embodiments of the present invention, the implementer may also customize the working scene of the bulldozer, and then input each environmental image into a trained scene classification model to output the working scene of the bulldozer at the corresponding acquisition moment; the training process and application thereof are already existing technologies well known to those skilled in the art and will not be elaborated on herein.

[0062] After determining the working scene of the bulldozer at each acquisition moment, the historical operation database of the bulldozer in each working scene is synchronously obtained. The historical operation database records the obstacle avoidance operation process of the bulldozer in each historical operation scene, including at least the obstacle avoidance mark of the bulldozer for each object in each historical environment image; that is, in a complete historical operation scene of the bulldozer, whether the bulldozer avoids obstacles while moving or working, and the obstacle avoidance mark is assigned to the object, providing a historical reference for obstacle avoidance in subsequent operation scenes.

[0063] It should be noted that the construction of the historical operation database can not only be based on the historical obstacle avoidance conditions of the current bulldozer's historical operation scenes, but can also be combined with the historical obstacle avoidance conditions in the historical operation scenes of other similar machines such as excavators, rollers and cranes to provide rich obstacle avoidance operation information for the construction of the historical operation database; its construction method is already an existing technology and will not be repeated here; the work logs of some unmanned bulldozers can be directly regarded as historical work databases.

[0064] Obstacle analysis module 102: used to obtain the initial obstacle avoidance coefficient of each obstacle type object in the corresponding work scene at each acquisition moment based on the appearance and obstacle avoidance marking of each obstacle type object in the historical operation database; obtain the dynamic obstruction index of each object at each acquisition moment based on the changes in the environmental image at adjacent acquisition moments; at each acquisition moment, obtain the obstacle avoidance coefficient of each object in the corresponding environmental image based on the dynamic obstruction index of each object and the initial obstacle avoidance coefficient of the obstacle type to which each object belongs.

[0065] Considering that the historical operation database of each work scenario can provide the bulldozer with historical obstacle avoidance references, that is, the bulldozer's obstacle avoidance habits or tendencies for each obstacle type in similar work scenarios, thus helping to assess the necessity of obstacle avoidance for each obstacle type in the bulldozer's working environment at each collection moment;

[0066] Therefore, in the embodiment of the present invention, in the corresponding working scenario at each acquisition moment, the initial obstacle avoidance coefficient of each obstacle type object is obtained according to the appearance and obstacle avoidance marking of each obstacle type object in the historical operation database; the initial obstacle avoidance coefficient is based on the historical obstacle avoidance reference in similar working scenarios, and preliminarily reflects the possibility of the bulldozer avoiding the obstacle type object, preparing for the subsequent distinction between obstacle targets and working targets.

[0067] Preferably, in one embodiment of the present invention, a target type is first determined for analysis. Considering that in a large number of similar historical operation scenes, if the number of times the bulldozer avoids an object of the target type is relatively high, it means that the bulldozer has a greater probability of avoiding the target type object at the corresponding collection time of the similar operation scene. Considering that the obstacle avoidance marking of the target type object in the historical operation scene is also affected by the frequency of occurrence, when the total frequency of occurrence is lower and the frequency of obstacle avoidance marking is relatively higher, it means that the confidence level of the bulldozer's obstacle avoidance possibility for the target type object is higher. Therefore, the method for obtaining the initial obstacle avoidance coefficient includes:

[0068] In the historical operation database corresponding to the work scenario at each collection moment, any obstacle type is used as the target type;

[0069] The ratio of the number of obstacle avoidance marks for the target type object to the total number of obstacle avoidance marks for all objects is used as the obstacle avoidance ratio of the target type object; the ratio of the number of obstacle avoidance marks for the target type object to the number of occurrences of the target type object is used as the obstacle avoidance frequency of the target type object;

[0070] The obstacle avoidance ratio and frequency are integrated to obtain the historical obstacle avoidance parameters of the target type object. The negative correlation mapping result of the occurrence frequency of the target type object is used as the confidence weight, and the historical obstacle avoidance parameters are weighted using the confidence weight. The weighted result is used as the initial obstacle avoidance parameter of the target type object.

[0071] As an example, in the two ratios corresponding to the obstacle avoidance ratio and the obstacle avoidance frequency, the number of obstacle avoidance markings of the target type object is used as the numerator; the obstacle avoidance ratio and the obstacle avoidance frequency are multiplied and combined to obtain the historical obstacle avoidance parameters; the larger the obstacle avoidance ratio and the obstacle avoidance probability, the more obvious the historical obstacle avoidance reference provided, which means that the bulldozer has a greater possibility of avoiding the target type object and the larger the historical obstacle avoidance parameter; the frequency of occurrence of the target type object in the historical operation database is inversely calculated to perform negative correlation normalization to obtain the confidence weight; then the confidence weight is multiplied and combined with the historical obstacle avoidance parameter to obtain the initial obstacle avoidance parameter of the target type object; the target type is changed to obtain the initial obstacle avoidance parameter for each obstacle type object.

[0072] Considering that the dynamic changes of objects may cause certain obstacles, for example, when a previously stationary pile of materials collapses or rolls, it may affect the movement or operation of a bulldozer. Therefore, it is even more important to assess the difficulty or urgency of obstacle avoidance caused by dynamic objects. In addition, considering the difference in environmental image changes between adjacent acquisition moments can help assess the changes of objects and thus help evaluate their dynamic obstruction index.

[0073] Therefore, the embodiment of the present invention obtains the dynamic obstruction index of each object at each acquisition time based on the changes in the environmental image at adjacent acquisition times. The dynamic obstruction index reflects the possibility of obstacles caused by the dynamic changes of the object, and prepares for the subsequent accurate evaluation of the obstacle avoidance coefficient in combination with the initial obstacle avoidance coefficient.

[0074] Preferably, in one embodiment of the present invention, considering that the greater the difference between the environmental image corresponding to each acquisition moment and the previous adjacent acquisition moment, the greater the environmental change, the greater the possibility of object change, and the greater the possibility of obstacle threat caused; considering that the position difference of the same object between two adjacent environmental images can reflect its movement, thereby helping to assess its obstruction to the movement or operation of the bulldozer; the dynamic change of the object and the obstruction possibility are comprehensively evaluated to comprehensively assess the dynamic obstruction situation of each object; therefore, the method for obtaining the dynamic obstruction index includes:

[0075] See also Figure 2 , which shows a flow chart of a method for obtaining a dynamic obstruction index provided by an embodiment of the present invention, specifically comprising:

[0076] Step S201 : performing frame difference between the environment image corresponding to each acquisition moment and the previous adjacent acquisition moment, and obtaining the dynamic change index of each acquisition moment according to the total number of pixels that have changed between the same positions.

[0077] As an example, first, an image coordinate system is constructed with the center point of each environmental image as the origin. Between each acquisition moment and the previous adjacent acquisition moment, the total number of pixels that have changed at the same position coordinates between the corresponding environmental images is obtained based on the frame difference method, that is, the total number of pixels whose grayscale value difference is not 0. Then the total number is directly used as the dynamic change index of each acquisition moment relative to the previous adjacent acquisition moment.

[0078] Step S202: Obtain a point cloud image at each acquisition moment and obtain the position coordinates of each object in the corresponding point cloud image; match the objects in the point cloud image corresponding to each acquisition moment with those in the previous adjacent acquisition moment, and obtain the movement direction of the corresponding object based on the difference in the position coordinates of the matched objects.

[0079] Considering that the point cloud image can provide the geometric information and orientation information of the object in three-dimensional space, it is convenient to evaluate the movement direction of the object between adjacent acquisition moments.

[0080] As an example, objects in each environment image are matched with objects in the point cloud image at the corresponding acquisition time. The matching method is well known to those skilled in the art and will not be described in detail. Then, the position coordinates of the geometric center, such as the center of mass, of each object can be obtained based on the corresponding point cloud image.

[0081] The centroid of the matching object at each acquisition moment corresponds to the position coordinates of the pixel point relative to the bulldozer to obtain the position coordinates of the object's centroid. The centroid position coordinates of each object at each acquisition moment are subtracted from the centroid position coordinates of the matching object at the previous acquisition moment to obtain the movement vector of each object, and its movement direction can be further determined.

[0082] It should be noted that since the lidar is located on the bulldozer and the acquisition frequency is 10 times per second, the possibility of the bulldozer's orientation changing between adjacent acquisition moments is extremely low, and thus will not affect the matching and position coordinate evaluation of objects in the point cloud images at adjacent acquisition moments.

[0083] Step S203 : obtaining a dynamic threat parameter of each object based on the difference between the moving direction of each object and the moving direction of the bulldozer; fusing the dynamic threat parameter with the dynamic change index to obtain a dynamic obstruction index of each object at each acquisition moment.

[0084] As an example, the absolute value of the cosine of the angle between the moving direction of each object and the moving direction of the bulldozer is added to a very small non-zero positive parameter such as 0.001, and then the reciprocal operation is performed. The reciprocal value is used as the dynamic threat parameter. The smaller the absolute value of the cosine of the angle between the two moving directions, the more likely it is that the two directions will intersect, and the greater the obstacle threat posed by the moving object to the movement or operation of the bulldozer. The dynamic threat parameter is then multiplied and fused with the dynamic change index to obtain the dynamic obstruction index of each object at each collection moment.

[0085] After obtaining each object's dynamic obstruction index at each acquisition moment, this embodiment of the present invention can further combine the initial obstacle avoidance coefficient of each object's obstacle type to obtain the obstacle avoidance coefficient for each object in the corresponding environment image. The obstacle avoidance coefficient ultimately reflects the likelihood that the object is an obstacle avoidance target, facilitating subsequent obstacle avoidance operations.

[0086] Preferably, in one embodiment of the present invention, the method for obtaining the obstacle avoidance coefficient includes:

[0087] At each acquisition moment, the dynamic obstruction index of each object is normalized, and the normalized value is used as the obstacle weight. The obstacle weight is used to weight the initial obstacle avoidance coefficient of the object of the obstacle type to which the corresponding object belongs, and the normalized result of the weighted result is used as the obstacle avoidance coefficient of the corresponding object.

[0088] As an example, the dynamic obstruction index of each object is first linearly normalized to obtain the obstacle weight, and then the obstacle weight of each object is multiplied by the initial obstacle avoidance coefficient of the obstacle type object to which the object belongs. The product is linearly normalized to obtain the obstacle avoidance coefficient of the corresponding object.

[0089] Obstacle avoidance planning module 103: used to determine the target type of the object according to the obstacle avoidance coefficient of each object at each acquisition moment, and plan the obstacle avoidance operation in real time.

[0090] Preferably, in one embodiment of the present invention, considering that the greater the obstacle avoidance coefficient, the greater the possibility that the corresponding object is an obstacle target; therefore, the method of determining whether the object is an obstacle target based on the obstacle avoidance coefficient of each object and planning an obstacle avoidance path includes:

[0091] In the environmental image at each acquisition moment, objects with an obstacle avoidance coefficient greater than a preset coefficient threshold are regarded as obstacle targets, and the remaining objects are regarded as operation targets;

[0092] Path planning for obstacle avoidance operations is performed based on all obstacle targets and all operation targets in the environment image.

[0093] Among them, in a preferred embodiment of the present invention, the preset coefficient threshold is 0.5, and the implementer can also customize it, so as to determine all obstacle targets and work targets in the environmental image at the corresponding acquisition time, where the obstacle targets are objects that need to be avoided, and the work targets are objects that need to be cleared; then the target types of objects in the point cloud image are distinguished and labeled accordingly, so as to facilitate subsequent path planning for obstacle avoidance operations.

[0094] In one embodiment of the present invention, after determining the target type of the object in the point cloud image, the morphological parameters of each object are further obtained, and risk assessment and obstacle avoidance requirements are determined in combination with the motion state of the bulldozer. For example, a nearest operating target is first determined, and an appropriate path planning algorithm such as the Dijkstra algorithm is used to plan an obstacle avoidance path for the bulldozer so that the bulldozer can smoothly reach the operating target to perform operations. When the bulldozer is moving or operating, environmental images at each acquisition moment will be collected to analyze changes in the surrounding environment and the dynamic conditions of the objects, so as to continuously adjust the obstacle avoidance strategy and path normalization to ensure that the bulldozer can avoid obstacles in real time and accurately in a complex dynamic environment, thereby achieving safe and efficient operations.

[0095] It should be noted that path planning and controlling the bulldozer to move and operate according to the obstacle avoidance operation path are both existing technologies. Some unmanned bulldozers can already automatically perform obstacle avoidance operations according to the obstacle avoidance path, and the specific obstacle avoidance operation planning process will not be repeated.

[0096] An embodiment of the present invention further proposes a bulldozer intelligent operation method based on machine vision. The method can utilize the above-mentioned bulldozer intelligent obstacle avoidance system based on machine vision to plan the obstacle avoidance operation of the bulldozer in real time.

[0097] In summary, during bulldozer operation, the present invention obtains an initial obstacle avoidance coefficient for each obstacle type based on the appearance and obstacle avoidance marking of each obstacle type in similar historical work scenes corresponding to each acquisition moment; obtains a dynamic obstruction index for each object at each acquisition moment based on changes in the environmental image at adjacent acquisition moments; and then obtains the obstacle avoidance coefficient for each object in the corresponding environmental image at each acquisition moment, thereby determining whether the object is an obstacle target and planning obstacle avoidance operations in real time. The present invention preliminarily assesses the obstacle avoidance possibility for each obstacle type object based on the historical obstacle avoidance situation of the bulldozer in similar work scenes, and then comprehensively assesses the obstacle avoidance coefficient of each object based on the dynamic information of each object, thereby accurately distinguishing between obstacle avoidance targets and work targets to plan obstacle avoidance operations, thereby improving the intelligent obstacle avoidance effect and work efficiency of the bulldozer.

[0098] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0099] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. The bulldozer intelligent obstacle avoidance system based on machine vision is characterized by: The system comprises: Scene acquisition module: used to obtain environmental images at each acquisition moment during bulldozer operation and mark the obstacle type of each object in the image; determine the working scene of the bulldozer at each acquisition moment and obtain a historical operation database for each working scene. The historical operation database includes at least the obstacle avoidance mark of each object in each historical environmental image; Obstacle analysis module: used to obtain the initial obstacle avoidance coefficient of each obstacle type object in the corresponding work scene at each acquisition moment based on the appearance and obstacle avoidance marking of each obstacle type object in the historical operation database; obtain the dynamic obstruction index of each object at each acquisition moment based on the changes in the environmental image at adjacent acquisition moments; at each acquisition moment, obtain the obstacle avoidance coefficient of each object in the corresponding environmental image based on the dynamic obstruction index of each object and the initial obstacle avoidance coefficient of the obstacle type to which each object belongs; Obstacle avoidance planning module: used to determine the target type of the object according to the obstacle avoidance coefficient of each object at each collection moment, and plan the obstacle avoidance operation in real time.

2. The machine vision-based intelligent obstacle avoidance system for bulldozers according to claim 1, characterized in that: The method for determining the working scenario includes: In each environment image, the environment complexity coefficient is obtained based on the shape, size, physical hardness, and proportion of each obstacle type in the environment image; According to the environmental complexity coefficient of each environmental image, the working scene of the bulldozer at the corresponding acquisition moment is determined.

3. The machine vision-based intelligent obstacle avoidance system for bulldozers according to claim 2, characterized in that: The method for obtaining the environmental complexity coefficient includes: For each obstacle type, the area ratio of all objects in the environment image is used as the obstacle density, the average area of ​​all objects is used as the shape size, and the product of the shape size and the preset physical hardness is used as the obstacle parameter; The obstacle density and the obstruction parameter of each obstacle type object are combined to obtain an environmental complexity coefficient.

4. The machine vision-based intelligent obstacle avoidance system for bulldozers according to claim 2, characterized in that: The method for obtaining the working scene of the bulldozer at the corresponding collection time according to the environmental complexity coefficient includes: When the environment complexity coefficient is less than or equal to a first preset threshold, it is determined to be in a low-complexity working scene; When the environment complexity coefficient is greater than a first preset threshold and less than a second preset threshold, it is determined to be a medium-complexity working scene; When the environment complexity coefficient is greater than or equal to a second preset threshold, it is determined to be in a high-complexity work scene; The first preset threshold is smaller than the second preset threshold.

5. The machine vision-based intelligent obstacle avoidance system for bulldozers according to claim 1, characterized in that: The method for obtaining the initial obstacle avoidance coefficient includes: In the historical operation database corresponding to the work scenario at each collection moment, any obstacle type is used as the target type; The ratio of the number of obstacle avoidance marks for the target type object to the total number of obstacle avoidance marks for all objects is used as the obstacle avoidance ratio of the target type object; the ratio of the number of obstacle avoidance marks for the target type object to the number of occurrences of the target type object is used as the obstacle avoidance frequency of the target type object; The obstacle avoidance ratio and the obstacle avoidance frequency are integrated to obtain historical obstacle avoidance parameters of the target type object; a negative correlation mapping result of the occurrence frequency of the target type object is used as a confidence weight, the historical obstacle avoidance parameters are weighted using the confidence weight, and the weighted result is used as the initial obstacle avoidance parameter of the target type object.

6. The machine vision-based intelligent obstacle avoidance system for bulldozers according to claim 1, characterized in that: The method for obtaining the dynamic obstacle index includes: Perform frame difference between the environment image at each acquisition moment and the previous adjacent acquisition moment, and obtain the dynamic change index at each acquisition moment based on the total number of pixels that have changed between the same positions; Obtain a point cloud image at each acquisition moment and the position coordinates of each object in the corresponding point cloud image; match the objects in the point cloud image corresponding to each acquisition moment with those in the previous adjacent acquisition moment, and obtain the movement direction of the corresponding object based on the difference in the position coordinates of the matched objects; According to the difference between the moving direction of each object and the moving direction of the bulldozer, the dynamic threat parameter of each object is obtained; and the dynamic threat parameter is integrated with the dynamic change index to obtain the dynamic obstruction index of each object at each collection moment.

7. The machine vision-based intelligent obstacle avoidance system for bulldozers according to claim 1, characterized in that: The method for obtaining the obstacle avoidance coefficient includes: At each acquisition moment, the dynamic obstruction index of each object is normalized, and the normalized value is used as the obstacle weight. The initial obstacle avoidance coefficient of the object of the obstacle type to which the corresponding object belongs is weighted using the obstacle weight, and the normalized result of the weighted result is used as the obstacle avoidance coefficient of the corresponding object.

8. The machine vision-based intelligent obstacle avoidance system for bulldozers according to claim 1, characterized in that: The method of determining whether an object is an obstacle target according to the obstacle avoidance coefficient of each object and planning an obstacle avoidance path includes: In the environmental image at each acquisition moment, objects with the obstacle avoidance coefficient greater than a preset coefficient threshold are regarded as obstacle targets, and the remaining objects are regarded as operation targets; Path planning for obstacle avoidance operations is performed based on all obstacle targets and all operation targets in the environment image.

9. The machine vision-based intelligent obstacle avoidance system for bulldozers according to claim 8, characterized in that: The preset coefficient threshold is 0.

5.

10. The bulldozer intelligent operation method based on machine vision is characterized by: The method utilizes the machine vision-based intelligent obstacle avoidance system for a bulldozer according to any one of claims 1 to 9 to plan the obstacle avoidance operation of the bulldozer in real time.

Images